SEARCH · Search NASA

Results for “data quality metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

All sources NASA NTRS DOE OSTI

DOE OSTI2026

Machine Learning-Based Anomaly Detection for PMT Data Quality Monitoring in the SBN and DUNE

Maintaining high-quality detector data is essential for achieving the scientific objectives of the Short-Baseline Neutrino (SBN) Program at Fermilab. Current data quality monitoring (DQM) procedures rely primarily on threshold-based metrics and manual inspection of detector monitoring plots, making the detection of subtle or gradually developing anomalies both time-consuming and dependent on expert interpretation. This project developed and evaluated a machine-learning workflow for automatically identifying anomalous photomultiplier tube (PMT) channels in the Short-Baseline Near Detector (SBND) using optical-hit amplitude data. A Python-based analysis program was developed to process ROOT files, extract statistical features describing individual PMT amplitude distributions, and generate feature vectors for anomaly detection. These features were used to train an Isolation Forest model using data representing normal detector operation. The trained model was subsequently applied to independent detector runs to identify channels exhibiting statistically unusual behavior relative to the learned reference response. To support expert interpretation, the workflow generated complementary diagnostic products, including anomaly score distributions, normalized amplitude comparisons, decision-tree visualizations, and principal component analysis (PCA) projections. This project demonstrated the feasibility of integrating unsupervised machine learning into detector data-quality monitoring and developed a complete workflow for automated PMT performance assessment to aid expert-driven review. Beyond its technical contributions, the VFP appointment fostered a research collaboration between Aurora University and Fermilab and provided direct workforce development benefits by training the visiting faculty member in detector-scale machine-learning methods that are now being incorporated into undergraduate coursework and research. The methodology developed here provides a foundation for future applications to ProtoDUNE and other liquid argon time projection chamber (LArTPC) detectors, contributing to ongoing efforts to improve detector reliability, reduce manual monitoring requirements, and enable scalable data quality monitoring for future large-scale neutrino experiments, including the Deep Underground Neutrino Experiment (DUNE).

Colón Santana, Juan A. [Unlisted, US, IL]↗

DOE OSTI2022

Validation of seismic velocity models in southern California with full-waveform simulations

SUMMARY Crustal seismic velocity models provide essential information for many applications including earthquake source properties, simulations of ground motion and related derivative products. We present a systematic workflow for assessing the accuracy of velocity models with full-waveform simulations. The framework is applied to four regional seismic velocity models for southern California: CVM-H15.11, CVM-S4.26, CVM-S4.26.M01 that includes a shallow geotechnical layer, and the model of Berg et al. For each model, we perform 3-D viscoelastic wave propagation simulations for 48 virtual seismic noise sources (down to 2 s) and 44 moderate-magnitude earthquakes (down to 2 s generally and 0.5 s for some cases) assuming a minimum shear wave velocity of 200 m s–1. The synthetic waveforms are compared with observations associated with both earthquake records and noise cross-correlation data sets. We measure, at multiple period bands for well-isolated seismic phases, traveltime delays and normalized zero-lag cross-correlation coefficients between the synthetic and observed data. The obtained measurements are summarized using the mean absolute derivation of time delay and the mean correlation coefficient. These two metrics provide reliable statistical representations of model quality with consistent results in all data sets. In addition to assessing the overall (average) performance of different models in the entire study area, we examine spatial variations of the models’ quality. All examined models show good phase and waveform agreements for surface waves at periods longer than 5 s, and discrepancies at shorter periods reflecting small-scale heterogeneities and near-surface structures. The model performing best overall is CVM-S4.26.M01. The largest misfits for both body and surface waves are in basin structures and around large fault zones. Inaccuracies generated in these areas may affect tomography and model simulation results at other regions. The seismic velocity models for southern California can be improved by adding better resolved structural representations of the shallow crust and volumes around the main faults.

58 GEOSCIENCES↗

DOE OSTI2020

Generation and evaluation of synthetic patient data

Background: Machine learning (ML) has made a significant impact in medicine and cancer research; however, its impact in these areas has been undeniably slower and more limited than in other application domains. A major reason for this has been the lack of availability of patient data to the broader ML research community, in large part due to patient privacy protection concerns. High-quality, realistic, synthetic datasets can be leveraged to accelerate methodological developments in medicine. By and large, medical data is high dimensional and often categorical. These characteristics pose multiple modeling challenges. Methods: In this paper, we evaluate three classes of synthetic data generation approaches; probabilistic models, classification-based imputation models, and generative adversarial neural networks. Metrics for evaluating the quality of the generated synthetic datasets are presented and discussed. Results: While the results and discussions are broadly applicable to medical data, for demonstration purposes we generate synthetic datasets for cancer based on the publicly available cancer registry data from the Surveillance Epidemiology and End Results (SEER) program. Specifically, our cohort consists of breast, respiratory, and non-solid cancer cases diagnosed between 2010 and 2015, which includes over 360,000 individual cases. Conclusions: We discuss the trade-offs of the different methods and metrics, providing guidance on considerations for the generation and usage of medical synthetic data.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2022

A review of computing-based automated fault detection and diagnosis of heating, ventilation and air conditioning systems

We report faults in Heating, Ventilation, and Air Conditioning (HVAC) systems of buildings result in significant energy waste in building operation. With fast-growing sensing data availability and advancement in computing, computational modeling has demonstrated strong capability to detect and diagnose HVAC system faults, hence, ensuring efficient building operation. This paper comprehensively reviews the state-of-the-art computing-based fault detection and diagnosis (FDD) for HVAC systems. Overall, the reviewed computing-based FDD methods are classified as two major approaches: knowledge-based and data-driven approaches. We then identify multiple important topics, including data availability, training data size, data quality, approach generality, capability, interpretability, and required modeling efforts, along with corresponding metrics to summarize the most updated FDD development. Generally, the knowledge-based approaches are further divided as physics-based modeling, Diagnostic Bayesian Network, and performance indicator-based methods while data-driven approaches include supervised learning, unsupervised learning, and regression and statistics-based methods. State-of-the-art FDD development, remaining challenges, and future research directions are further discussed to push forward FDD in practice. Availability of fault data, capability of existing methods to deal with complex fault situations (such as simultaneous faults), modeling interpretability for data-driven methods, and required engineering efforts for physics-based methods are identified as remaining challenges in FDD development. Improving modeling fidelity and reducing modeling efforts are essential for applying physics-based methods in real buildings. Meanwhile, addressing fault data availability, increasing algorithm adaptability, and handling multiple faults are essential to further enhance the applicability of data-driven FDD approaches.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2022

Metrics as tools for bridging climate science and applications

In climate science and applications, the term “metric” is used to describe the distillation of complex, multifaceted evaluations to summarize the overall quality of a model simulation, or other data product, and/or as a means to quantify some response to climate change. Metrics provide insights into the fidelity of processes and outcomes from climate models and can assist with both differentiating models' representation of variables or processes and informing whether models are “fit for purpose.” Metrics can also provide a valuable reference point for co-production of knowledge between climate scientists and climate impact practitioners. Although continued metric developments enable model developers to better understand the impacts of decisions made in the model design process, metrics also have implications for the characterization of uncertainty and facilitating analyses of underlying physical processes. As a result, comprehensive evaluation with multiple metrics enhances usability of climate information by both scientific and stakeholder communities. Here, this paper presents examples of insights gained from the development and appropriate use of metrics, and provides examples of how metrics can be used to engage with stakeholders and inform decision-making.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2026

Surrogate Distributed Radiological Sources—Part III: Quantitative Distributed Source Reconstructions

In this third part of a multi-paper series, we present quantitative image reconstruction results from aerial measurements of eight different surrogate distributed gamma-ray sources on flat terrain. Here, we show that our quantitative imaging methods can accurately reconstruct the expected shapes, and, after appropriate calibration, the absolute activity of the distributed sources. We conduct several studies of imaging performance versus various measurement and reconstruction parameters, including detector altitude and raster pass spacing, data and modeling fidelity, and regularization type and strength. The imaging quality performance is quantified using various quantitative image quality metrics. Our results confirm the utility of point source arrays as surrogates for truly distributed radiological sources, and advance the quantitative capabilities of Scene Data Fusion gamma-ray imaging methods.

Airborne survey↗

DOE OSTI2024

PMU Data Quality and Sensor Health Monitoring

Phasor Measurement Units (PMUs) play a critical role in the evolution of the electric power industry by providing high-precision, real-time monitoring of essential power system metrics. However, effectively detecting abnormalities and critical events from PMU data is a complex task, complicated by intricate temporal patterns, a scarcity of labeled data for training algo- rithms, and constraints on online computational power. In this study, we apply TranAD, an innovative algorithm that combines transformer architectures with the refinement of adversarial learning, to both synthetic and real-world PMU datasets for developing a data quality and sensor online health monitoring platform for utilities. Our findings reveal that TranAD not only provides efficient detection and localization but also enhances the detail with which abnormalities are detected, marking a a significant step forward in the field of clean data acquisition processes for power system monitoring

deep neural network, machine learning (ML)↗

DOE OSTI2021

Scalable Programming Workflows for Validation of Quantum Computers

Hybrid quantum-classical workflows have become standard methods for executing variational algorithms and other quantum simulation techniques, which are key applications for noisy intermediate scale quantum (NISQ) computers. Validating these simulations is an important task which helps gauge the progress of quantum computer development, and classical simulation can serve as a tool to this end. Both exact and more scalable approximate methods with quantifiable error bounds can be used in validation tasks where the applicable metrics include the distance from a calculable ground truth, the quality of an error model fit to data, etc. Here we present a library extension that includes methods for validation of quantum simulations based on scalable hybrid workflows executable on high performance computers. We provide examples that use approximate methods based on tensor networks and stabilizer simulators to bound the error of quantum simulations on NISQ hardware.

Nguyen, Thien↗

DOE OSTI2023

Quality Assessment of Molten Salt Thermochemical Property Data

Recommendations are made to enhance user confidence in property values included in the Molten Salt Thermal Database-Thermochemical Properties (MSTDB-TC). Specifically, it is recommended that all available measured values be included with a transparent assessment of each to arrive at a preferred value for each salt composition in the database. The use of a quality assessment and ranking system is recommended for data and property values compiled in the MSTDB-TC that is analogous to the approach developed for data and property values included in the MSTDB-Thermophysical Properties (TP) database. The approach includes assessments of five features of data collection: the method used to measure the property value, calibration of devices used to make the measurements, the quantified uncertainty of the results, control of the environmental conditions, and determination of the salt composition. The quality of each facet is ranked as High, Moderate, or Incomplete based on objective metrics. The overall quality of the dataset is ranked based on rankings of the five data features and traceability of the reported property value, with rankings of A and B indicating the data and property value are deemed suitable for quantitative use, C indicating the value is computed and not validated and U indicating uncertainties restrict the recommended use to qualitative purposes. Sufficient information should be provided or cited within each data set included in the database to verify the reported property value. The recommended approach is suitable for assessing data already in the MSTDB-TC and screening new entries.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2022

Introduction to Special Section: Machine Learning for Image-based Geologic Interpretation

Image-based geological interpretation has been a labor-intensive and time-consuming process because it requires well-trained geoscientists to identify geological structures, features, and textures from various types of images. These images include scanning electron microscopic images, optical microscopic images, optical photos, resistivity images, seismic volumes, remote-sensing images, etc. With fast-evolving machine learning (ML) technology and computing power in recent decades, computers can achieve nearhuman-level to super-human-level performance with scalable high efficiency in the computer vision field. These technological revolutions facilitated image-based geological interpretation in petroleum exploration and production. For example, a fault picking method applied to 3-D seismic volume data using deep learning can achieve superior performance in comparison to conventional auto-picking methods. In addition, under the new normal of low oil prices, the petroleum industry seeks cost-effective strategies such as automating traditionally labor-intensive processes. Nevertheless, the potential of applying ML to geological image interpretation is still facing a few key challenges including data scarcity, data distribution, poor data and/or label quality, data leakage, learning algorithms, model architecture, training methodologies, testing and evaluation metrics, hyper-parameters optimization, model drift, production deployment, and the like.

58 GEOSCIENCES↗

DOE OSTI2021

International collaboration framework for the calculation of performance loss rates: Data quality, benchmarks, and trends (towards a uniform methodology)

Abstract The IEA PVPS Task 13 group, experts who focus on photovoltaic performance, operation, and reliability from several leading R&D centers, universities, and industrial companies, is developing a framework for the calculation of performance loss rates of a large number of commercial and research photovoltaic (PV) power plants and their related weather data coming across various climatic zones. The general steps to calculate the performance loss rate are (i) input data cleaning and grading; (ii) data filtering; (iii) performance metric selection, corrections, and aggregation; and finally, (iv) application of a statistical modeling method to determine the performance loss rate value. In this study, several high‐quality power and irradiance datasets have been shared, and the participants of the study were asked to calculate the performance loss rate of each individual system using their preferred methodologies. The data are used for benchmarking activities and to define capabilities and uncertainties of all the various methods. The combination of data filtering, metrics (performance ratio or power based), and statistical modeling methods are benchmarked in terms of (i) their deviation from the average value and (ii) their uncertainty, standard error, and confidence intervals. It was observed that careful data filtering is an essential foundation for reliable performance loss rate calculations. Furthermore, the selection of the calculation steps filter/metric/statistical method is highly dependent on one another, and the steps should not be assessed individually.

14 SOLAR ENERGY↗

DOE OSTI2026

Connecting ambient toxicity testing with community-level responses of benthic macroinvertebrates in an impacted stream in East Tennessee, USA

Single-species laboratory toxicity tests are a standard tool for evaluating potential impairment of freshwater systems; however, it remains uncertain how well they reflect community-level impacts in natural environments. This study presents a multi-decadal dataset (2005-2025) pairing ambient toxicity testing with macroinvertebrate surveys along Bear Creek on the Oak Ridge Reservation (Tennessee, USA) downstream of an industrial complex to assess the ability of laboratory tests using stream water to track community-level effects. Biannual three-brood Ceriodaphnia dubia tests from 2005 to 2025 often showed reduced reproduction at select sites. Integrating water quality data showed strong positive correlations between sublethal toxicity and specific conductance. Macroinvertebrate diversity metrics, family-level occurrence, and densities were also associated with conductance and contemporaneous C. dubia responses. Laboratory-measured sublethal toxicity was a stronger indicator of macroinvertebrate change than conductance alone, although responses varied among sites and seasons. At the site with the highest diversity, densities and richness of Ephemeroptera, Plecoptera, Trichoptera (EPT) and non-EPT taxa were significantly related to C. dubia reproduction, with greater toxicity corresponding to lower diversity. At the family level, some pollution-tolerant taxa were more prevalent and at higher densities during periods of sublethal toxicity, while some sensitive families were absent or reduced. These patterns may reflect site-specific mixtures of acute and chronic stressors, with laboratory toxicity tests more effectively capturing short-term impacts. Overall, these multi-decadal observations suggest that laboratory toxicity tests can help track water-quality changes linked to shifts in aquatic community diversity, despite variable responses reflecting the complexity of dynamic stressors in this impacted freshwater system.

Stevenson, Louise [ORNL] (ORCID:0000000349679897)↗

DOE OSTI2026

RTN-117: Image Calibration and Instrument Signal Removal for the First Year of the LSST

The NSF-DOE Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) requires calibration products that provide uniform, stable, and accurate photometric and astrometric performance across the 3.2 gigapixel focal plane and throughout the 10-year survey. This paper details the algorithms and workflows used to produce instrument calibrations and remove instrumental artifacts for Data Preview 2 (DP2)---the first end-to-end processing demonstration using on-sky data with the LSST Camera (LSSTCam). We describe the verification, acceptance, and certification framework used to assess calibration quality and quantify residual systematics. We show baseline metrics on calibrated science images to evaluate the robustness of the calibration and instrument signature removal (ISR) data processing pipelines for DP2. Finally, we summarize the known limitations observed in DP2 production and outline expected algorithmic improvements for the first public LSST data release (Data Release~1, DR1).

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2023

Know Your Space: Inlier and Outlier Construction for Calibrating Medical OOD Detectors

This software offers methods and functions for training calibrated out-of-distribution detectors for medical image classification tasks. It includes functionalities for training, synthesizing data augmentations, calibration, and out-of-distribution detection. Developed using PyTorch, this software is compatible with standard neural network architectures used for imaging data. Additionally, it provides capabilities to compute evaluation metrics for assessing the performance and quality of the detectors.

Narayanaswamy, VivekSivaraman↗

DOE OSTI2024

DECIDER

This software offers methods and functions for building failure detectors for deep image classification models with the aid of vision-language models and LLMs. It includes functionalities for training baseline image classifiers, debiasing classifiers using vision-language models and LLMs, evaluating failure between models along with baselines. Developed using PyTorch, this software is compatible with standard neural network architectures used for imaging data. Additionally, it provides capabilities to compute evaluation metrics for assessing the performance and quality of the detectors.

Narayanaswamy, Vivek Sivaraman↗

DOE OSTI2024

Towards Full Field-of-View Fourier Ptychography for Extreme Ultraviolet Microscope

We evaluate various Fourier ptychographic microscopy (FPM) reconstruction algorithms using both simulated and experimental data acquired from an Extreme Ultraviolet (EUV, 13.5 nm wavelength) microscope. We specifically focus on the algorithms' ability to robustly address field-dependent aberrations, which enables increased spatial resolution and quantitative phase imaging across an expanded field of view. We systematically compare the algorithms' performance under aberrations for a single zoneplate imaging system, utilizing Fourier Ring Correlation (FRC) as a systematic metric for assessing reconstruction quality. Furthermore, we explore the impact of systematic errors on the reconstruction of experimental data, aiming to increase the effective field of view by 25-fold, from the nominal 5x5 um2 diffraction-limited area. Additionally, our evaluation incorporates innovative FPM-adjacent methodologies, including the Angular Ptychographic Imaging with Closed-form method (APIC), for reconstructing EUV images.

Gu, Chaoying↗

DOE OSTI2025

Accuracy-Based Annotation Quality Score (ABAQS) v1.0

Assessing genome annotation quality is crucial for downstream analyses, but current methods are inadequate for eukaryotes. We present Accuracy-Based Annotation Quality Score (ABAQS), a novel, minimal-data-driven method that comprehensively assesses annotation quality. ABAQS evaluates multiple factors, including genome completeness, gene model validity, and protein profile accuracy, outperforming other metrics like BUSCO and PSAURON. We applied ABAQS to over 2500 eukaryotic genomes and showed its robustness and effectiveness in evaluating genome annotation quality, making it a valuable tool for researchers working with genomic data. ABAQS reveals significant variation in annotation quality and highlights the importance of filtering in improving annotation quality and accuracy.

Haridas, Sajeet [Lawrence Berkeley National Labora↗

DOE OSTI2020

Clustering Analysis of Commercial Vehicles Using Automatically Extracted Features from Time Series Data

Standard of practice approaches to time series cluster analysis involve careful feature engineering, often utilizing expert input to tune and select features by hand. In many cases, expert input may not be readily available, or there may not yet exist a community consensus on the ideal features for a given application. This paper compares the results of several cluster analysis methods, using both hand selected features and those extracted automatically, when applied to large geospatial time series telematics data from commercial trucking fleets. The impacts of feature selection, dimensionality reduction, and choice of clustering algorithm on the quality of clustering results are explored. Results from this analysis confirm prior results that domain agnostic features are competitive with the hand engineered features with respect to clustering quality metrics. These results also provide new insight into the most successful strategies for identifying structure in large unstructured vehicle telematics data, and suggest that time series clustering using automatic feature extraction can be an effective approach to extract structure from large scale geospatial time series data in cases when hand selected features are not available.

33 ADVANCED PROPULSION SYSTEMS↗