Search NASA⌕ Search

SEARCH · Search NASA

Results for “data quality metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

FedFleet 2024: Fleet Data Quality - Understanding and Improving Your Agency's FAST Fleet Data Quality Metric? [Slides]

This presentation presents an overview of how the team supporting the Federal Automotive Statistical Tool (FAST) project for DOE and GSA view and assess the quality of fleet data submissions from federal agencies. It also discusses ways in which agencies can improve the quality of their fleet data and of their fleet data submissions. FAST is a web-based information system sponsored by GSA's Office of Government-wide Policy and DOE's Federal Energy Management Program to collect information about the US federal government's fleet of motor vehicles; FAST is developed, maintained, and supported by DOE's Idaho National Laboratory (INL).

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2020

Data Qualification Framework

Proposed Data Qualification System Background Originally developed as a data qualification system for the Bioenergy Feedstock Library Provide data users a means of assessing data quality based on user driven quality metrics. Give users information regarding common data quality metrics to determine and assess the data quality. Potential Use Same type of quality assessment framework could be potentially applied to other databases and/or datasets for common assessment. Provide metric for assessment of appropriateness of data for various uses.

09 BIOMASS FUELS↗
DOE OSTI2024

Characterization of Non-Science Grade DESI CCDs and Creating Additional ICARUS Monitoring Metrics for Data Quality

DESI, otherwise known as the Dark Energy Spectroscopic Instrument, is an astronomical project measuring millions of optical spectra to characterize Dark Energy. The survey has been running for 3 years. My work delves into the data taking/analysis side of the DESI CCDs (Charge-Coupled Devices) which will help in swapping out broken CCDs on the instrument. Additionally, I worked on making new metrics for ICARUS, Imaging Cosmic and Rare Underground Signals for their monitoring of run data. This paper will talk about the types of characterizations done for DESI and additional metrics for data quality and tools used for ICARUS monitoring.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

Livewire: Automatic Annotations

Diogenes processes datasets to provide data quality metrics for the Livewire platform and creates standardized data dictionaries from data annotations. Diogenes needs data annotations that clearly outline thenformat and organization of the data. It also relies on the type, class, and unit of each data piece for comprehensive analysis, which it cannot determine independently. The Annotation Tool significantly reduces the time needed to create annotations for Diogenes by generating data annotations with the correct formatting and content. It also employs machine learning and hard-coded models to automatically annotate data class, quality type, and data units.

33 - ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2024

Collaborative Research: Properties and Dynamics of the Shallow Crust (Final Report)

Ground motions recorded at one location are often extrapolated to nearby regions within a given radius. Here, we explore the appropriateness of spatial extrapolation using data from seven small aperture seismic network deployments in southern California. Six of these deployments are linear arrays of 4-13 stations, and one is a 2D array of 13 stations at Pinyon Flats Observatory. The spatial footprint array diameters are 3 km or less, and each array was operational for a year or more. From our base catalog (M2.5+ earthquakes; 4038 events; September 2010 - June 2023), automated methods remove temporally overprinted waveforms from nearby events (< 5 km) in quick succession (< 5 min) and data with nonviable waveforms. These 200 samples per second data are filtered at 0.5-25 Hz and must have signal-tonoise ratios (SNRs) of 2.5+. Peak ground acceleration (PGA) and peak ground velocity (PGV) are derived individually from the maximum absolute values of each of the 3-component waveforms (vertical, northsouth, and east-west). Five of the seven arrays traverse the San Jacinto fault, and two do not. Ground motion observations are compared with theoretical estimates from Abrahamson et al., 2014. On average, arrays deployed within and across fault zones consistently record ground motions above theoretical expectations, whereas off-fault arrays record ground motions at or slightly below theoretical expectations. We attribute these differences to site conditions because these trends prevail for the full data suites. For each network and each individual channel, the coefficient of variation indicates that the standard deviations are ~30±6% of the mean. Exploring relative ground motion contributions from all three channels (ternary plots), as expected, most data show that vertical ground motions are attenuated compared to horizontal ones. However, this is not always the case for RA array data, where vertical motions can be ~2-3 times larger than horizontal motions for select events near Cahuilla, CA. These anomalously high vertical motions are focal mechanism-related. These results suggest that ternary plots created using only a small amount of data can be used as a data quality metric and a tool to find anomalous features in three-component data.

58 GEOSCIENCES↗
DOE OSTI2026

LSST Scheduler & Progress Monitoring

The NSF-DOE Vera C. Rubin Observatory scheduling team is providing a set of tools for monitoring the scheduler and progress on the Legacy Survey of Space and Time (LSST). These slides provide an overview of available pages and assorted figures on them: the night summary provides an overview of a completed night of observing, including maps of exposures and plots of data quality metrics; the pre-night briefing, an analysis of a simulation of an upcoming night; and a handful of figures that provide information on long-term progress toward LSST survey goals.

H.Neilsen, Eric, Jr. [Fermilab] (ORCID:00000002735↗
DOE OSTI2025

Data Readiness for AI: A 360-Degree Survey

Artificial Intelligence (AI) applications critically depend on data. Poor-quality data produces inaccurate and ineffective AI models that may lead to incorrect or unsafe use. Evaluation of data readiness is a crucial step in improving the quality and appropriateness of data usage for AI. R&D efforts have been spent on improving data quality. However, standardized metrics for evaluating data readiness for use in AI training are still evolving. In this study, we perform a comprehensive survey of metrics used to verify data readiness for AI training. This survey examines more than 140 papers published by ACM Digital Library, IEEE Xplore, journals such as Nature, Springer, and Science Direct, and online articles published by prominent AI experts. This survey aims to propose a taxonomy of data readiness for AI (DRAI) metrics for structured and unstructured datasets. We anticipate that this taxonomy will lead to new standards for DRAI metrics that would be used for enhancing the quality, accuracy, and fairness of AI training and inference.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Fast and efficient identification of anomalous galaxy spectra with neural density estimation

ABSTRACT Current large-scale astrophysical experiments produce unprecedented amounts of rich and diverse data. This creates a growing need for fast and flexible automated data inspection methods. Deep learning algorithms can capture and pick up subtle variations in rich data sets and are fast to apply once trained. Here, we study the applicability of an unsupervised and probabilistic deep learning framework, the probabilistic auto-encoder, to the detection of peculiar objects in galaxy spectra from the SDSS survey. Different to supervised algorithms, this algorithm is not trained to detect a specific feature or type of anomaly, instead it learns the complex and diverse distribution of galaxy spectra from training data and identifies outliers with respect to the learned distribution. We find that the algorithm assigns consistently lower probabilities (higher anomaly score) to spectra that exhibit unusual features. For example, the majority of outliers among quiescent galaxies are E+A galaxies, whose spectra combine features from old and young stellar population. Other identified outliers include LINERs, supernovae, and overlapping objects. Conditional modelling further allows us to incorporate additional information. Namely, we evaluate the probability of an object being anomalous given a certain spectral class, but other information such as metrics of data quality or estimated redshift could be incorporated as well. We make our code publicly available.

Böhm, Vanessa↗
DOE OSTI2024

Characterization of Non-Science Grade DESI CCDs and Creating Additional ICARUS Monitoring Metrics

DESI, otherwise known as the Dark Energy Spectroscopic Instrument, is a tool to measure optical spectra from the effects of dark energy on the universe and has been running for 3 years. My work delves into the data taking/analysis side of the DESI CCDs which will help in swapping out broken CCDs on the instrument. Additionally, I worked on making new metrics for ICARUS, Imaging Cosmic and Rare Underground Signals for their monitoring of run data. This paper will talk about the types of characterizations done for DESI and some metrics for data quality and tools used for ICARUS monitoring.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2022

Reanalysis of crackle perception data using logistic and logarithmic fits and sound quality metrics

Mathematical models of human auditory perception (sound quality metrics) are explored for the purpose of fitting human subject data from a prior published study. A variety of linearizing transforms and techniques are employed. Ultimately, a model is identified that predicts 98.9% of the variance in average human subject ratings of crackliness of jet noise waveforms.

Swift, Stephen Hales↗
DOE OSTI2023

BigNeuron: a resource to benchmark and predict performance of algorithms for automated tracing of neurons in light microscopy datasets

BigNeuron is an open community bench-testing platform with the goal of setting open standards for accurate and fast automatic neuron tracing. We gathered a diverse set of image volumes across several species that is representative of the data obtained in many neuroscience laboratories interested in neuron tracing. Here, we report generated gold standard manual annotations for a subset of the available imaging datasets and quantified tracing quality for 35 automatic tracing algorithms. The goal of generating such a hand-curated diverse dataset is to advance the development of tracing algorithms and enable generalizable benchmarking. Together with image quality features, we pooled the data in an interactive web application that enables users and developers to perform principal component analysis, t-distributed stochastic neighbor embedding, correlation and clustering, visualization of imaging and tracing data, and benchmarking of automatic tracing algorithms in user-defined data subsets. The image quality metrics explain most of the variance in the data, followed by neuromorphological features related to neuron size. Furthermore, we observed that diverse algorithms can provide complementary information to obtain accurate results and developed a method to iteratively combine methods and generate consensus reconstructions. The consensus trees obtained provide estimates of the neuron structure ground truth that typically outperform single algorithms in noisy datasets. However, specific algorithms may outperform the consensus tree strategy in specific imaging conditions. Finally, to aid users in predicting the most accurate automatic tracing results without manual annotations for comparison, we used support vector machine regression to predict reconstruction quality given an image volume and a set of automatic tracings.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Improved quality metrics for association and reproducibility in chromatin accessibility data using mutual information

Correlation metrics are widely utilized in genomics analysis and often implemented with little regard to assumptions of normality, homoscedasticity, and independence of values. This is especially true when comparing values between replicated sequencing experiments that probe chromatin accessibility, such as assays for transposase-accessible chromatin via sequencing (ATAC-seq). Such data can possess several regions across the human genome with little to no sequencing depth and are thus non-normal with a large portion of zero values. Despite distributed use in the epigenomics field, few studies have evaluated and benchmarked how correlation and association statistics behave across ATAC-seq experiments with known differences or the effects of removing specific outliers from the data. Here, we developed a computational simulation of ATAC-seq data to elucidate the behavior of correlation statistics and to compare their accuracy under set conditions of reproducibility. Using these simulations, we monitored the behavior of several correlation statistics, including the Pearson’s R and Spearman’s ρ coefficients as well as Kendall’s τ and Top–Down correlation. We also test the behavior of association measures, including the coefficient of determination R 2 , Kendall’s W, and normalized mutual information. Our experiments reveal an insensitivity of most statistics, including Spear man’s ρ, Kendall’s τ, and Kendall’s W, to increasing differences between simulated ATAC-seq replicates. The removal of co-zeros (regions lacking mapped sequenced reads) between simulated experiments greatly improves the estimates of correlation and association. After removing co-zeros, the R 2 coefficient and normalized mutual information display the best performance, having a closer one-to-one relationship with the known portion of shared, enhanced loci between simulated replicates. When comparing values between experimental ATAC-seq data using a random forest model, mutual information best predicts ATAC-seq replicate relationships. Collectively, this study demonstrates how measures of correlation and association can behave in epigenomics experiments. We provide improved strategies for quantifying relationships in these increasingly prevalent and important chromatin accessibility assays.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Review of capture cross sections relevant for intentional nuclear forensics

The NA-22 Intentional Forensics Venture is developing a system for tagging nuclear fuel using various methods of information encoding. One of the main methods under development is the insertion of isotopically enriched tracers into the fuel. In order to aid in the understanding of the neutronic performance of these taggants, we assess the quality of the nuclear data underpinning simulations, which are driven by the neutron-capture cross sections. We present these cross sections of naturally occurring isotopes of the elements provided in the neutron sublibrary of the planned ENDF/B-VIII.1 Feb. 2023 library release. We make this assessment using a rubric designed for this effort, which quantifies orthogonal features related to the overall quality. The quality metric highlights 6 aspects: experimental data, resonance evaluations, integral metrics, covariances, fission products, and documentation. We focus on energy ranges relevant for reactor applications. We also discuss additional sources for new, high-quality cross-section data that may be utilized on the time scale of the venture, including existing global data, new experiments, and computational methods. Finally, overall outlook is presented with conclusions.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

Validating automated resonance evaluation with synthetic data

The integrity and precision of nuclear data are crucial for a broad spectrum of applications, from national security and nuclear reactor design to medical diagnostics, where the associated uncertainties can significantly impact outcomes. A substantial portion of uncertainty in nuclear data originates from the subjective biases in the evaluation process, a crucial phase in the nuclear data production pipeline. Recent advancements indicate that automation of certain routines can mitigate these biases, thereby standardizing the evaluation process and enhancing reproducibility. This research aims to provide a methodology, framework, and metrics for the validation of automated nuclear data evaluation software leveraging high-quality synthetic data that closely mimic real experimental observables. An introduced error metric provides a scale and intuitive measure of the evaluation quality by quantifying the estimate’s accuracy and performance across the specified energy range. Synthetic data provides access to experimental observables and underlying resonance parameters, enabling comparison of different evaluations. The methodology is demonstrated using Ta-181 isotope data in the resolved resonance region. The Automated Resonance Identification Subroutine (ARIS), which operates without prior resonance information, was used to test and showcase the framework’s capabilities utilizing the proposed error metrics. The results demonstrate the effectiveness of the proposed approach and framework for optimizing software parameters and testing hypotheses through “what-if” controlled experiments, such as modifying assumptions about experimental conditions or average resonance parameters.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2022

A Data Quality-Aware Framework to Reliably Forecast Photovoltaic Generation and Consumer Load for an Improved Resilience of Microgrids

Photovoltaic (PV) power and consumer load forecasting plays a critical role to ensure operational resilience of the electric grid. Most data-driven forecasting algorithms rely heavily on the continuous availability of good quality data for periodic training and validation. When deployed at the grid’s edge, prolonged disruptions to communications during extreme events degrade data quality. Factors such as missing observations, epistemic uncertainties, data drift, and concept drift are manifestations of data quality that impact the generalization of such field-deployed forecasting models. Currently, there exists no mechanism in the literature to dynamically switch between models under varying degrees of data quality as quantified by certain metrics for each factor highlighted above. This paper addresses this shortcoming by conceptually introducing a data qualityaware framework for reliable PV generation and consumer load forecasting. The framework’s design incorporates components of missing values, divergence tests, and continuous monitoring of generalization performance to detect changes in data quality caused by communications disruptions and trigger specific classes of forecasting models grouped under three use cases (UC1- UC3). As a first step towards validating this framework, real data collected from an actual field microgrid system is used to demonstrate the viability of the three use cases. Results show that the performance is the best in UC1 with an unadjusted R-square value of 0.954, followed by 0.939 for UC2 and 0.757 for UC3.

Sundararajan, Aditya↗
DOE OSTI2023

Visualization Quality Assessment

Understanding how inaccuracies in visualizations affect users’ perception and understanding of scientific data is hard. Inaccuracies in visualizations are quite common and could arise from a range of sources such as errors in the original dataset arising from compression artifacts, errors in the capturing device, noise during transmission of the data, effects due to the algorithm being used to convert data to visualization images, images generated from neural networks, and sources we have yet to discover. Many image quality assessment metrics have been developed to quantify image errors. However, these are usually focused on “natural images” rather than visualizations of scientific data. Common image quality assessment metrics (IQAs) include MSE, PSNR, perceptual metrics such SSIM, FSIM as well as perceptual metrics using deep learning approaches. However, a critical part of understanding how errors are perceived by humans, and subsequently developing more accurate quality assessment metrics, is through user evaluation studies. The goal of this software is to develop a visualization quality assessment (VQA) process that will enable the generation of VQAs that can be used to quantify errors in scientific data visualizations. The VQA development process will include software to support user evaluation experimental design, analysis of visualization differences against standard quality metrics, and the ability to develop additional VQA metrics specific to scientific visualization images.

Grosset, Andre↗