Search NASA⌕ Search

SEARCH · Search NASA

Results for “metric standardization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

A call to standardize metrics for monitoring baleen whales near marine construction activities

Effective monitoring is necessary to protect marine mammal species during the construction of offshore infrastructure. The tools for detecting or monitoring marine mammals span traditional (e.g., visual observers, optical cameras), to newer (e.g., passive acoustic monitoring, infrared cameras, tags), and emerging (e.g., satellite imagery, environmental DNA, dimethyl sulfide concentration) technologies. Some are better suited for use during offshore development; however, peer-reviewed literature does not typically evaluate and report on the performance of these various technologies. We define a minimum set of metrics related to efficacy (i.e., confusion matrix, precision and recall, probability of missed mitigation), detection range (i.e., maximum and reliable detection range, spatial resolution), and data delivery (i.e., detection latency, system reliability, temporal resolution) that we recommend are needed to assess the utility of monitoring technologies for this purpose. Following a literature review of relevant studies, we highlight which publications reported these metrics and used multiple technologies to compare relative performance. We also emphasize the benefits of multi-modal approaches and recommend performance assessments through modeling or large-scale collaborative field testing. These metrics will standardize data collection, reporting, and analysis; promote consistent and comparable results; and foster collaboration among developers, regulatory agencies, and scientists. This may lead to the co-development of technology that achieves multiple goals, has greater application, and can answer research questions while collecting data to fulfill permitting requirements. These metrics may also inform decisions on what systems regulatory agencies might consider using and reduce monitoring costs, which is critical to support the marine sector's rapid growth alongside marine mammal conservation.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2021

Standardizing Performance Metrics for Building-Level Electrical Distribution Systems

Building-level electrical distribution systems comprise a myriad of current-carrying equipment, conversion devices, and protection devices that deliver power from the utility or local distributed energy resources to end-use building loads. Electric power has traditionally been generated, transmitted, and distributed in alternating current (AC). However, the last decade has seen a significant increase in the integration of native direct current (DC) equipment that has elevated the importance of DC distribution systems. Numerous studies have comparatively examined the performance of various electrical distribution systems in buildings but have failed to achieve uniform conclusions, primarily because of a lack of consistent and analogous performance evaluation methods. This paper aims to fill this gap by providing a standard set of metrics and measurement boundaries to consistently evaluate the performance of AC, DC, or hybrid AC/DC electrical distribution systems. The efficacy of the proposed approach is evaluated on a representative medium-sized commercial office building model with AC distribution and an equivalent hybrid AC/DC and DC distribution model, wherein the AC distribution model is concluded to be the most efficient. The simulation results show variation in computed metrics with different selected boundaries that verify the effectiveness of the proposed approach in ensuring consistent computation of the performance of building-level electrical distribution systems. This paper provides an initial set of guidelines for building energy system stakeholders to adopt appropriate solutions, thus leading to more efficient energy systems.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2021

Reliability metrics and their management implications for open pond algae cultivation

The prevalence of contaminating organisms in outdoor algae cultivation, with the often-associated dramatic crop failures, necessitates the need for metrics describing production system reliability. Standard metrics for algae cultivation reliability are critically needed to be able to map improvements in operational parameters, but do not currently exist. In this work, we present a set of standard metrics including mean time to failure (MTTF) and mean time between failures (MTBF) as the basis for the calculation of pond failure rate (FR) and reliability coefficient (RC). Metrics associated with the numbers of contaminating organisms such as abundance ratio (AR), prevalence of infection (PI), and mean intensity of infection (MII) are also relevant. These metrics, based on measured experimental values of outdoor pond performance during the Algae Testbed Public-Private Partnership (ATP3) Unified Field Studies (UFS), provide a basis for quantifying pond failure and provide insight into potential pond management and contaminant mitigation strategies. From these reliability metrics applied to this dataset, we are able to link operational parameters, such as harvest frequency and inoculum source, to pond reliability for different algae strains and seasons, and from an assessment of AR, provide contamination thresholds beyond which a culture may be unrecoverable. Ultimately, the implementation of widely-practiced and simple-to-calculate algae pond reliability metrics calculated from rapid and easy to collect data or observations will reduce risk and uncertainty in large-scale algae deployment and aid in the development of integrated pest management (IPM) strategies.

09 BIOMASS FUELS↗
DOE OSTI2023

Specificity in plant-mycorrhizal fungal relationships: prevalence, parameterization, and prospects

Species interactions exhibit varying degrees of specialization, ranging from generalist to specialist interactions. For many interactions (e.g., plant-microbiome) we lack standardized metrics of specialization, hindering our ability to apply comparative frameworks of specificity across niche axes and organismal groups. Here, we discuss the concept of plant host specificity of arbuscular mycorrhizal (AM) fungi and ectomycorrhizal (EM) fungi, including the predominant theories for their interactions: Passenger, Driver, and Habitat Hypotheses. We focus on five major areas of interest in advancing the field of plant-mycorrhizal fungal host specificity: phylogenetic specificity, host physiology specificity, functional specificity, habitat specificity, and mycorrhizal fungal-mediated plant rarity. Considering the need to elucidate foundational concepts of specificity in this globally important symbiosis, we propose standardized metrics and comparative studies to enhance our understanding. We also emphasize the importance of analyzing global mycorrhizal data holistically to draw meaningful conclusions and suggest a shift toward single-species analyses to unravel the complexities underlying these associations.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Data Readiness for AI: A 360-Degree Survey

Artificial Intelligence (AI) applications critically depend on data. Poor-quality data produces inaccurate and ineffective AI models that may lead to incorrect or unsafe use. Evaluation of data readiness is a crucial step in improving the quality and appropriateness of data usage for AI. R&D efforts have been spent on improving data quality. However, standardized metrics for evaluating data readiness for use in AI training are still evolving. In this study, we perform a comprehensive survey of metrics used to verify data readiness for AI training. This survey examines more than 140 papers published by ACM Digital Library, IEEE Xplore, journals such as Nature, Springer, and Science Direct, and online articles published by prominent AI experts. This survey aims to propose a taxonomy of data readiness for AI (DRAI) metrics for structured and unstructured datasets. We anticipate that this taxonomy will lead to new standards for DRAI metrics that would be used for enhancing the quality, accuracy, and fairness of AI training and inference.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Open Source Evaluation Framework for Solar Forecasting

The Solar Forecast Arbiter is an open-source evaluation framework for solar forecasting. The framework enables evaluations of solar irradiance, solar power, and net-load forecasts that are impartial, repeatable and auditable. The Solar Forecast Arbiter addresses stakeholder-informed use cases including evaluation of forecast skill, comparisons to reference data sets, private forecast trials, and evaluation of probabilistic forecast skill. The framework includes a data validation toolkit, reference data sources, data privacy protocols, and benchmark forecast capabilities for intra-hour and day ahead forecast horizons. Reports and metrics communicate the relative merits of the test and benchmark forecasts. The reports are created from standardized templates and include graphics for qualitatively evaluating deterministic and probabilistic forecasts and standard metrics for quantitatively evaluating forecasts. The Solar Forecast Arbiter is designed to support all solar forecasting stakeholders, including Solar Forecasting 2 Topic Area 2 and Topic Area 3 teams.

14 SOLAR ENERGY↗
DOE OSTI2022

Measures of quasisymmetry for stellarators

Quasisymmetric stellarators are an attractive class of optimised magnetic confinement configurations. The property of quasisymmetry (QS) is in practice limited to be approximate, and thus the construction requires measures that quantify the deviation from the exact property. In this paper we study three measure candidates used in the literature, placing the focus on their origin and a comparison of their forms. The analysis shows clearly the lack of universality in these measures. As these metrics do not directly correspond to any physical property (except when exactly quasisymmetric), optimisation should employ additional physical metrics for guidance. It is suggested that close to QS minima, one should treat QS metrics through inequality constraints so that additional physics metrics dominate optimisation. The impact of different quasisymmetric measures on optimisation is presented through an example, for which the standard metric that weights the asymmetric Fourier modes of the field magnitude appears to perform best.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2020

Quantifying Dynamic On-State Resistance of GaN HEMTs for Power Converter Design via a Survey of Low and High Voltage Devices

Designing and optimizing high switching frequency, ultra-efficient converters requires detailed knowledge of the behavior and parasitic parameters for both active and passive components. Recently, wide bandgap transistors have enabled simultaneous increases in both switching frequency and efficiency due to higher maximum operating junction temperature limits, lower dc on-state resistances, and reduced parasitic inductances and capacitances. Yet, the early acceptance of gallium-nitride (GaN) switches was plagued by detrimental dynamic on-state resistance effects. This complex phenomenon for GaN devices is characterized by deviations in on-state resistance from dc operating characteristics based on design choices such as the magnitude and duration of both voltage and current stress, switching mode, and junction temperature. While device manufacturers have made improvements compared to early generation devices, experimental evidence from a survey of commercial GaN transistors highlight measurable change in on-state resistance still exists due to variations in voltage stress during hard-switching operation. After sharing insights for obtaining low noise measurements, an analysis method along with two metrics are proposed to characterize dynamic on-state resistance measurements for power electronics designers. Furthermore, quantifying the performance of GaN devices with standardized metrics facilitates a fair comparison between different GaN device technologies during converter development, enables manufacturing qualification for GaN switches, and provides a benchmark to catalyze improvement for the next generation of GaN device development.

14 SOLAR ENERGY↗
DOE OSTI2022

Capsule network-based semantic segmentation model for thermal anomaly identification on building envelopes

Thermography technology is widely used to inspect thermal anomalies in building façade systems. Computer vision-based techniques provide opportunities to autonomously detect such heat anomalies to significantly improve the efficiency of decision-making for building envelope retrofitting and maintenance. Here, in this work, we propose a novel Capsule Network-based deep learning model – CapsLab – that detects and identifies thermal anomalies by semantic segmentation. CapsLab is built based on our proposed prediction-tuning capsule (PT-Capsule) layer. Different from a traditional capsule layer, which consists of part-whole transformation and capsule-routing process, the proposed layer is composed of a prediction and tuning process, which helps decreasing the number of model parameters significantly. While the applicability of traditional Capsule Networks (CapsNets) has been limited to simpler tasks and smaller datasets due to their scalability issue, we can leverage the lightweight of the proposed PT-Capsule layer, and apply it to the semantic segmentation task. In this work, we also employ our previously presented performance metric, referred to as the Anomaly Identification Metric (AIM) (Kakillioglua et al. 2021), to evaluate the segmentation outputs. Traditional performance metrics do not accurately reflect the true performance of the segmentation models in thermal anomaly identification due to the high subjectivity in the annotation process and higher overlap ratio sensitivity of the standard metrics. AIM, on the other hand, is robust to these drawbacks. Experimental results show, both qualitatively and quantitatively, that our proposed segmentation method can effectively segment the thermal anomalies. Specifically, our model provides 9.38% and 13.53% improvements over the baseline model – DeepLabV3+ – based on traditional mIoU score and the AIM score, respectively, while requiring less model parameters and less computation at the same time. In addition, the scores that the AIM metric generates better align with the scores provided by building performance experts.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

EVs@Scale Next-Gen Profiles - Fleet Utilization 2023

As U.S. fleet operators begin transitioning to electric vehicles (EVs), critical questions arise regarding how to manage this shift without disrupting fleet operations or placing undue stress on the electric grid. A major challenge for fleets is maintaining effective operational schedules while accommodating charging requirements, particularly with high-power charging (HPC) infrastructure, which presents grid stability concerns for utilities. Proposed solutions such as charging substations, megawatt charging systems (MCS), and smart charge management systems (SCMS) offer potential pathways forward, but their effectiveness depends on alignment with real-world fleet behavior and operational constraints. This report investigates the charging and utilization behavior of EV and EVSE fleets actively employing HPC technologies by conducting detailed case study analyses based on telematics data. A suite of predefined metrics—covering charging, routing, and other operational behaviors—is developed to evaluate the impact of fleet activities on grid infrastructure and identify opportunities for optimization. Results highlight variations in charging behavior across fleets, such as weekday versus weekend usage, diurnal charging trends, and the role of operational predictability in enabling SCMS effectiveness. While SCMS can help lower costs and improve energy efficiency for fleets with stable schedules, they may be insufficient for fleets with highly variable or long-haul operations, which may require more robust solutions like MCS. Visualization of aggregated hourly energy metrics reveals that while fleet behaviors are diverse, there are common temporal patterns that could inform infrastructure planning and energy management. These insights emphasize the need for fleet-specific charging strategies that minimize grid impact while supporting reliable fleet operations. Additionally, the report underscores the broader economic stakes of electrification, particularly in high-value markets such as freight, where misaligned transitions could stall EV adoption. By examining current EV and EVSE fleet deployments using predetermined standardized metrics, this study offers a foundation for developing technologies and operational frameworks that support scalable, grid-compatible electrification across a variety of fleet types while establishing a baseline understanding of operational behaviors. In doing so, we aim to ensure that future charging solutions reflect actual fleet needs and grid constraints—an essential step toward maintaining operational continuity and achieving a successful transition to electric fleet operations.

Charging↗
DOE OSTI2023

Technical Report on Waveform Fit Metrics for Global Models

The new WAVEFORMS Initiative in the Ground-based Nuclear Detonation Detection (GNDD) program includes an increased emphasis on the development of Earth models and methods to predict entire seismic and acoustic waveforms more accurately. In general, this increased emphasis is predicated on the need to better characterize seismic events and provide improved model-based discrimination between event types including earthquakes and explosions. More specifically, while current moment tensor inversion methods tend to work well for larger events (M>~4) using tuned 1-D Earth models, the development of state-of-the-art 3-D models and methods is required for the prediction of shorter period waves over large areas for discrimination of smaller events. There is no standard metric for model-based waveform prediction accuracy used in the waveform modeling/inversion community. However, there are several popular waveform misfit definitions; and minimizing the corresponding objective functions is the goal of waveform inversion. Some example misfit definitions employed for adjoint waveform tomography include measures of simple travel time differences (e.g. Tape et al., 2010), cross-correlation travel time differences (e.g. Luo and Schuster, 1991), multi-taper frequency dependent methods (e.g. Lei et al., 2020), time-frequency phase misfit functions (e.g. Fichtner 2010; Rodgers et al., 2022), normalized cross-correlation methods (e.g. Tao et al., 2018), and others. In some cases, these misfit definitions also involve complicated weighting schemes and summations over multiple frequency bands making it difficult to duplicate the misfit measurement with alternative models and datasets. Although each of the misfit definitions mentioned above are useful for developing waveform models, the actual misfit values are not usually meaningful outside of a given project, model, and/or dataset. Therefore, it is difficult to understand and communicate model performance for predicting waveforms and comparing to other models and/or new model iterations with a different dataset. Therefore, there is a need for a generalized method for evaluating overall model performance that is independent from the specific misfit chosen to develop the waveform models that is also intuitive and meaningful. In this report, we describe a new metric we refer to as ‘Percent of Correlated Signal’. The following sections describe and demonstrate the metric with a case study event and a more rigorous test using a random selection of globally distributed events. While the focus here is on global tomography models, the metric is meant to applicable to regional ‘wiggle-for-wiggle’ waveform models/studies as well.

58 GEOSCIENCES↗
DOE OSTI2025

EVs@Scale Next-Gen Profiles - Fleet Utilization 2024

As part of the U.S. Department of Energy’s EVs@Scale initiative, the Next-Gen Profiles (NGP) project provides a comprehensive, data-driven analysis of electric vehicle (EV) and electric vehicle supply equipment (EVSE) operations across real-world fleet deployments. This paper presents findings from the NGP’s Fleet Utilization study, which investigates operational behavior and asset usage across seventeen EV fleets and two EVSE fleets, encompassing a wide range of vehicle types and use cases. Data collected from diverse sources—varying in format and temporal resolution—are first reformatted into a unified structure. From this harmonized dataset, a suite of rigorously defined performance metrics is calculated at an hourly cadence, enabling consistent cross-comparison of charging, routing, and other key operational behaviors. Amid rapidly increasing EV adoption and growing demands for energy-efficient fleet operations, the analysis reveals clear utilization trends—including diurnal and weekly activity cycles, differences in short versus long charging session dependencies, and route-specific energy usage patterns. These findings highlight the need for tailored infrastructure strategies and the deployment of advanced energy management systems, such as Distributed Energy Resource Management Systems (DERMS) and Site Energy Management Systems (SEMS), which can optimize charging schedules and mitigate peak loads. By leveraging anonymized, harmonized datasets and standardized metrics, this study offers critical insights into fleet behavior and performance, providing a foundation to improve operational efficiency, reduce costs, and enable the scalable deployment of electrified transportation.

Wells, Landon↗
DOE OSTI2021

Quantitative Evaluation of Autonomous Driving in CARLA

There has been a great deal of recent advancements in end-to-end imitation and reinforcement learning for self-driving vehicles. Despite this, there is a severe lack of standardized metrics for evaluating the performance of autonomous self-driving agents. Existing metrics are generally lacking in their ability to capture a wide range of driving behaviors and compare the severity of different failure cases. In this work, we introduce the Quantitative Evaluation for Driving metric, or QED, which assigns a quantitative score from 0-100 that captures the quality of driving for any driving agent. Our QED metric assesses different aspects of driving behavior including the ability to stay in the center of the lane, avoid weaving and erratic behavior, follow the speed limit, and avoid collisions, and it can be used under a wide range of driving scenarios. To show the effectiveness of our QED metric, we compare the scores generated by QED against scores assigned by human evaluators on a total of 30 different drivers and 6 different towns in the CARLA driving simulator. In ``easy'' evaluation scenarios, where it is relatively straightforward to distinguish better drivers from worse drivers, QED attains an average Pearson correlation of 0.96 and average Spearman correlation of 0.97 when compared against human evaluators. In ``hard'' evaluation scenarios, where it is far more ambiguous how to rank/score different types of bad driving behavior, QED attains an average Pearson correlation of 0.82 and average Spearman correlation of 0.75 when compared against human evaluators, which are both slighter higher than when we compare human evaluators against each other. While QED may not capture every characteristic that defines good driving, we consider it an important foundation for reproducibility and standardization in the community.

Gao, Shang↗
DOE OSTI2020

Certifying the Performance of Fixative Technologies under Open Air Demolition Activities for D and D - 20096

Department of Energy (DOE) facilities undergoing deactivation and decommissioning (D and D) activities struggle with safety concerns of residual radioactive contamination leftover after gross decontamination efforts have concluded. The positive effects of implementing fixative technologies for decommissioning and maintenance is known across the DOE complex. However, there is no standardized metric in place that quantifies the performance of these technologies such as incombustible fixative platforms that immobilize the contamination in a solid polymer material under normal operating conditions or when exposed to stressors. The primary focus of this research is to provide the empirical data necessary to properly characterize the effects of implementing fixative technologies on mitigating the release of contamination. In addition, this effort has been used to identify and establish uniform testing protocols to quantify operational parameters of fixating platforms and properly credit such systems under a variety of operational conditions and stressors that can arise during open air demolition efforts (e.g. impact, water, thermal, etc.). The established protocols will be provided to the ASTM International E10.03 Subcommittee on Radiological Protection for Decontamination and Decommissioning of Nuclear Facilities and Components for the potential of formal standardization. Attributes to be addressed during testing are material compromise from impact stress, thermal stress for incombustible fixative materials, and immobilization factors (how well an immobilizing platform is capable of retaining fixated material during thermal and impact stressors). In developing these test methods, a greater understanding of the material's behavior under anticipated and unanticipated events, such as decommissioning activities and contingency events as outlined in the Basis of Interim Operations (BIO), can be successfully characterized. There are several key components of the experimental methodology that is essential in empirically certifying fixative technologies. A surrogate contaminant with a unique signature has been utilized with the implementation of a uniform contamination process of test coupons that is quantifiable and replicable for direct fixative comparisons. A modular test chamber has also been used to apply the various stressors of interest. Collection of released contamination that includes airborne and resettled contaminant particles and analyzed using mass spectrometry. This research directly supports DOE complex-wide concerns in terms of final disposition of nuclear facilities and the considerations made during their time between ceasing facility operation and final disposition. The results of this effort aims to provide a standardized road map and logical decision flow to better facilitate fixative technologies into D and D activities required to achieve the desired facility end state. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2024

Tau Positron Emission Tomography for Predicting Dementia in Individuals With Mild Cognitive Impairment

An accurate prognosis is especially pertinent in mild cognitive impairment (MCI), when individuals experience considerable uncertainty about future progression. To evaluate the prognostic value of tau positron emission tomography (PET) to predict clinical progression from MCI to dementia. This was a multicenter cohort study with external validation and a mean (SD) follow-up of 2.0 (1.1) years. Data were collected from centers in South Korea, Sweden, the US, and Switzerland from June 2014 to January 2024. Participant data were retrospectively collected and inclusion criteria were a baseline clinical diagnosis of MCI; longitudinal clinical follow-up; a Mini-Mental State Examination (MMSE) score greater than 22; and available tau PET, amyloid-β (Aβ) PET, and magnetic resonance imaging (MRI) scan less than 1 year from diagnosis. A total of 448 eligible individuals with MCI were included (331 in the discovery cohort and 117 in the validation cohort). None of these participants were excluded over the course of the study. Exposures included Tau PET, Aβ PET, and MRI. Positive results on tau PET (temporal meta–region of interest), Aβ PET (global; expressed in the standardized metric Centiloids), and MRI (Alzheimer disease [AD] signature region) was assessed using quantitative thresholds and visual reads. Clinical progression from MCI to all-cause dementia (regardless of suspected etiology) or to AD dementia (AD as suspected etiology) served as the primary outcomes. The primary analyses were receiver operating characteristics. In the discovery cohort, the mean (SD) age was 70.9 (8.5) years, 191 (58%) were male, the mean (SD) MMSE score was 27.1 (1.9), and 110 individuals with MCI (33%) converted to dementia (71 to AD dementia). Only the model with tau PET predicted all-cause dementia (area under the receiver operating characteristic curve [AUC], 0.75; 95% CI, 0.70-0.80) better than a base model including age, sex, education, and MMSE score (AUC, 0.71; 95% CI, 0.65-0.77; P = .02), while the models assessing the other neuroimaging markers did not improve prediction. In the validation cohort, tau PET replicated in predicting all-cause dementia. Compared to the base model (AUC, 0.75; 95% CI, 0.69-0.82), prediction of AD dementia in the discovery cohort was significantly improved by including tau PET (AUC, 0.84; 95% CI, 0.79-0.89; P < .001), tau PET visual read (AUC, 0.83; 95% CI, 0.78-0.88; P = .001), and Aβ PET Centiloids (AUC, 0.83; 95% CI, 0.78-0.88; P = .03). In the validation cohort, only the tau PET and the tau PET visual reads replicated in predicting AD dementia. In this study, tau-PET showed the best performance as a stand-alone marker to predict progression to dementia among individuals with MCI. This suggests that, for prognostic purposes in MCI, a tau PET scan may be the best currently available neuroimaging marker.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

Energy and power quality measurement for electrical distribution in AC and DC microgrid buildings

Today's selection of DC microgrid buildings features a diverse set of electrical topologies and turnkey solutions, each with specific design trade-offs and optimizations. Designers desperately need standardized metrics and procedures for measurement and verification (M&V) to analyze and compare the advantages of each DC solution to traditional AC building networks. This work develops M&V procedures for quantifying and comparing the energy efficiency and power quality in buildings. To calculate full-building efficiency, this work introduces the measurement-informed modeling method, a procedure that develops and refines a building's energy model with metered data. To quantify power quality, this work defines a new voltage quality index that applies to both AC and DC buildings. This article describes the equipment, instrumentation, and operation necessary to calculate the efficiency and power quality. It then demonstrates these methods with a set of field tests. We report these M&V procedures can ultimately be used to compare and improve the efficiency and power quality of various DC topologies.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2025

Leveraging large language models to automate the identification of healthcare access barriers for veterans

Objective: To develop and evaluate an automated system for identifying healthcare barriers focusing on transportation issues in veterans’ clinical notes using large language models (LLMs) and to assess the impact of different prompting strategies on classification performance and explanation consistency. Methods: We developed a hybrid system combining pattern matching for templated notes with LLM analysis for free-text notes. Using 2000 manually annotated clinical notes, we compared four prompting strategies (dual-role short, dual-role long, analysis-first, analysis-only) across Mistral-7B and Llama-3.1 models. We evaluated classification performance using standard metrics and assessed explanation consistency through embedding similarity analysis. Results: The analysis-first strategy achieved superior performance, with Mistral-7B reaching an F1 score of 0.914, outperforming traditional machine learning approaches (GBM: 0.786, BERT: 0.811). LLMs demonstrated higher explanation consistency within models (mean cosine similarity 0.887–0.908) compared to cross-model similarities (0.767–0.872). Pattern matching successfully handled 6.7% of templated notes deterministically. Mistral-7B showed greater internal consistency but higher abstention rates compared to Llama-3.1. Conclusion: Requiring LLMs to analyze evidence before classification improves both accuracy and explanation consistency for identifying transportation barriers in clinical notes. This approach enables automated barrier detection at scale while providing clinically relevant explanations, supporting both population-level healthcare planning and individual patient care decisions.

Healthcare access barriers↗
DOE OSTI2024

Temperature and Composition Dependence Modeling of Viscosity and Electrical Conductivity of Low-Activity Waste Glass Melts

The development of models that accurately relate the properties of a glass melt to its temperature and composition is important for glass formulation, melter control, and modeling the melt flow, refractory corrosion, and production rate. Using a database consisting of more than 4,000 data points measured between 900 °C and 1250 °C for over 600 unique low-activity waste glass compositions, we developed models for the melt viscosity and electrical conductivity. Models based on the Gaussian process regression approach outperformed models based on the Vogel–Fulcher–Tammann equation according to four standard metrics and yielded reliable prediction intervals. The models found primarily linear effects between properties and individual components, except for the effect of the Na 2 O mass fraction on the electrical conductivity. The effects were found to be consistent with current theories on physical processes involved with those properties.

36 MATERIALS SCIENCE↗