Search NASA⌕ Search

SEARCH · Search NASA

Results for “data quality metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

BigNeuron: a resource to benchmark and predict performance of algorithms for automated tracing of neurons in light microscopy datasets

BigNeuron is an open community bench-testing platform with the goal of setting open standards for accurate and fast automatic neuron tracing. We gathered a diverse set of image volumes across several species that is representative of the data obtained in many neuroscience laboratories interested in neuron tracing. Here, we report generated gold standard manual annotations for a subset of the available imaging datasets and quantified tracing quality for 35 automatic tracing algorithms. The goal of generating such a hand-curated diverse dataset is to advance the development of tracing algorithms and enable generalizable benchmarking. Together with image quality features, we pooled the data in an interactive web application that enables users and developers to perform principal component analysis, t-distributed stochastic neighbor embedding, correlation and clustering, visualization of imaging and tracing data, and benchmarking of automatic tracing algorithms in user-defined data subsets. The image quality metrics explain most of the variance in the data, followed by neuromorphological features related to neuron size. Furthermore, we observed that diverse algorithms can provide complementary information to obtain accurate results and developed a method to iteratively combine methods and generate consensus reconstructions. The consensus trees obtained provide estimates of the neuron structure ground truth that typically outperform single algorithms in noisy datasets. However, specific algorithms may outperform the consensus tree strategy in specific imaging conditions. Finally, to aid users in predicting the most accurate automatic tracing results without manual annotations for comparison, we used support vector machine regression to predict reconstruction quality given an image volume and a set of automatic tracings.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2010

Evaluation of Dual Pressurized Rover Operations During Simulated Planetary Surface Exploration

Introduction: A pair of small pressurized rovers (Space Exploration Vehicles, or SEVs) is at the center of the Global Point-of-Departure architecture for future human planetary exploration. Simultaneous operation of multiple crewed surface assets should maximize productive crew time, minimize overhead, and preserve contingency return paths. Methods: A 14-day mission simulation was conducted in the Arizona desert as part of NASA?s 2010 Desert Research and Technology Studies (DRATS). The simulation involved two SEV concept vehicles performing geological exploration under varied operational modes affecting both the extent to which the SEVs must maintain real-time communications with mission control ("Continuous" vs. "Twice-a-Day") and their proximity to each other ("Lead-and-Follow" vs. "Divide-and-Conquer"). As part of a minimalist lunar architecture, no communications relay satellites were assumed. Two-person crews consisting of an astronaut and a field geologist operated each SEV, day and night, throughout the entire 14-day mission, only leaving via the suit ports to perform simulated extravehicular activities. Standard metrics enabled quantification of the habitability and usability of all aspects of the SEV concept vehicles throughout the mission, as well as comparison of the extent to which the operating modes affected crew productivity and performance. Practically significant differences in the relevant metrics were prospectively defined for the testing of all hypotheses. Results and Discussion: Data showed a significant 14% increase in available science time (AST) during Lead-and-Follow mode compared with Divide-and-Conquer, primarily because of the minimal overhead required to maintain communications during Lead-and-Follow. In Lead-and-Follow mode, there was a non-significant 2% increase in AST during Twice-a-Day vs. Continuous communications. Situational awareness of the other vehicle?s location, activities, and contingency return constraints were enhanced during Lead-and-Follow and Twice-a-Day communications modes due to line-of-sight and direct SEV-to-SEV communication. Preliminary analysis of Scientific Data Quality and Observation Quality metrics showed no significant differences between modes.

Abercromby, Andrew F. J.↗
NASA NTRSDate not supplied

Lucy Orbit Determination Performance From Launch Through EGA-1

Lucy is a 12-year long NASA Discovery-class mission which launched on October 16, 2021 and is currently on its way to the Jupiter-Trojan asteroids at the L4 and L5 Lagrange points. On its journey to the Trojan asteroids, Lucy will also fly by two main belt asteroids, and perform 3 Earth fly bys. This paper summarizes primarily the Orbit Determination (OD) aspect of the first year of the mission from launch through the first Earth fly by (EGA-1). Some challenges were encountered for OD as part of the +Y solar array not fully deploying, but these were resolved within the first year of flight. The high quality radio-metric data from the Deep Space Network including 2- and 3-way Doppler, sequential 2-way ranging, complemented by delta differential one-way ranging (∆DOR) has enabled the OD team to characterize the Solar Radiation Pressure force, resolve desaturation maneuvers, generate both maneuver reconstructions and accurate long-term predicted trajectories. This paper discusses the filter setup and strategies implemented, as well as the OD performance from launch, through cruise, and including EGA-1.

Jeroen L. Geeraert↗
DOE OSTI2023

Improved quality metrics for association and reproducibility in chromatin accessibility data using mutual information

Correlation metrics are widely utilized in genomics analysis and often implemented with little regard to assumptions of normality, homoscedasticity, and independence of values. This is especially true when comparing values between replicated sequencing experiments that probe chromatin accessibility, such as assays for transposase-accessible chromatin via sequencing (ATAC-seq). Such data can possess several regions across the human genome with little to no sequencing depth and are thus non-normal with a large portion of zero values. Despite distributed use in the epigenomics field, few studies have evaluated and benchmarked how correlation and association statistics behave across ATAC-seq experiments with known differences or the effects of removing specific outliers from the data. Here, we developed a computational simulation of ATAC-seq data to elucidate the behavior of correlation statistics and to compare their accuracy under set conditions of reproducibility. Using these simulations, we monitored the behavior of several correlation statistics, including the Pearson’s R and Spearman’s ρ coefficients as well as Kendall’s τ and Top–Down correlation. We also test the behavior of association measures, including the coefficient of determination R 2 , Kendall’s W, and normalized mutual information. Our experiments reveal an insensitivity of most statistics, including Spear man’s ρ, Kendall’s τ, and Kendall’s W, to increasing differences between simulated ATAC-seq replicates. The removal of co-zeros (regions lacking mapped sequenced reads) between simulated experiments greatly improves the estimates of correlation and association. After removing co-zeros, the R 2 coefficient and normalized mutual information display the best performance, having a closer one-to-one relationship with the known portion of shared, enhanced loci between simulated replicates. When comparing values between experimental ATAC-seq data using a random forest model, mutual information best predicts ATAC-seq replicate relationships. Collectively, this study demonstrates how measures of correlation and association can behave in epigenomics experiments. We provide improved strategies for quantifying relationships in these increasingly prevalent and important chromatin accessibility assays.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Review of capture cross sections relevant for intentional nuclear forensics

The NA-22 Intentional Forensics Venture is developing a system for tagging nuclear fuel using various methods of information encoding. One of the main methods under development is the insertion of isotopically enriched tracers into the fuel. In order to aid in the understanding of the neutronic performance of these taggants, we assess the quality of the nuclear data underpinning simulations, which are driven by the neutron-capture cross sections. We present these cross sections of naturally occurring isotopes of the elements provided in the neutron sublibrary of the planned ENDF/B-VIII.1 Feb. 2023 library release. We make this assessment using a rubric designed for this effort, which quantifies orthogonal features related to the overall quality. The quality metric highlights 6 aspects: experimental data, resonance evaluations, integral metrics, covariances, fission products, and documentation. We focus on energy ranges relevant for reactor applications. We also discuss additional sources for new, high-quality cross-section data that may be utilized on the time scale of the venture, including existing global data, new experiments, and computational methods. Finally, overall outlook is presented with conclusions.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

Validating automated resonance evaluation with synthetic data

The integrity and precision of nuclear data are crucial for a broad spectrum of applications, from national security and nuclear reactor design to medical diagnostics, where the associated uncertainties can significantly impact outcomes. A substantial portion of uncertainty in nuclear data originates from the subjective biases in the evaluation process, a crucial phase in the nuclear data production pipeline. Recent advancements indicate that automation of certain routines can mitigate these biases, thereby standardizing the evaluation process and enhancing reproducibility. This research aims to provide a methodology, framework, and metrics for the validation of automated nuclear data evaluation software leveraging high-quality synthetic data that closely mimic real experimental observables. An introduced error metric provides a scale and intuitive measure of the evaluation quality by quantifying the estimate’s accuracy and performance across the specified energy range. Synthetic data provides access to experimental observables and underlying resonance parameters, enabling comparison of different evaluations. The methodology is demonstrated using Ta-181 isotope data in the resolved resonance region. The Automated Resonance Identification Subroutine (ARIS), which operates without prior resonance information, was used to test and showcase the framework’s capabilities utilizing the proposed error metrics. The results demonstrate the effectiveness of the proposed approach and framework for optimizing software parameters and testing hypotheses through “what-if” controlled experiments, such as modifying assumptions about experimental conditions or average resonance parameters.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2022

A Data Quality-Aware Framework to Reliably Forecast Photovoltaic Generation and Consumer Load for an Improved Resilience of Microgrids

Photovoltaic (PV) power and consumer load forecasting plays a critical role to ensure operational resilience of the electric grid. Most data-driven forecasting algorithms rely heavily on the continuous availability of good quality data for periodic training and validation. When deployed at the grid’s edge, prolonged disruptions to communications during extreme events degrade data quality. Factors such as missing observations, epistemic uncertainties, data drift, and concept drift are manifestations of data quality that impact the generalization of such field-deployed forecasting models. Currently, there exists no mechanism in the literature to dynamically switch between models under varying degrees of data quality as quantified by certain metrics for each factor highlighted above. This paper addresses this shortcoming by conceptually introducing a data qualityaware framework for reliable PV generation and consumer load forecasting. The framework’s design incorporates components of missing values, divergence tests, and continuous monitoring of generalization performance to detect changes in data quality caused by communications disruptions and trigger specific classes of forecasting models grouped under three use cases (UC1- UC3). As a first step towards validating this framework, real data collected from an actual field microgrid system is used to demonstrate the viability of the three use cases. Results show that the performance is the best in UC1 with an unadjusted R-square value of 0.954, followed by 0.939 for UC2 and 0.757 for UC3.

Sundararajan, Aditya↗
NASA NTRS2018

An Overall Assessment of JPSS-2 VIIRS Radiometric Performance Based on Pre-Launch Testing

The Visible Infrared Imaging Radiometer Suite (VIIRS) on-board the second Joint Polar Satellite System (JPSS) completed its sensor level testing in February 2018. The JPSS-2 (J2) mission is scheduled to launch in 2022 and will be very similar to its two predecessor missions, the Suomi National Polar-orbiting Partnership (SNPP) mission, launched on 28 October 2011 and JPSS-1 (renamed NOAA-20) launched on 18 November 2017. VIIRS instrument has 22 spectral bands covering the spectrum between 0.4 and 12.6 μm: 14 reflective solar bands (RSB), 7 thermal emissive bands (TEB) and one day-night band (DNB). It is a cross-track scanning radiometer capable of providing global measurements through observations at two spatial resolutions, 375 m and 750 m at nadir for the imaging bands and moderate bands, respectively. This paper will provide an overview of J2 VIIRS characterization methodologies and calibration performance during the pre-launch testing phases performed by the National Aeronautics and Space Administration (NASA) VIIRS Characterization Support Team (VCST) to evaluate the at-launch baseline radiometric performance and generate the parameters needed to populate the sensor data record (SDR) Look-Up-Tables (LUTs). Our analysis results confirmed the good performance of J2 VIIRS, in general as good as previous VIIRS instruments and all non-compliances are expected to have low impact on data quality. Key sensor performance metrics include the signal to noise ratio (SNR), radiance dynamic range, reflective and emissive bands calibration performance, polarization sensitivity, spectral performance, response versus scan-angle (RVS) and scattered light response. A set of performance metrics generated during the pre-launch testing program will be compared to both the SNPP and JPSS-1 VIIRS sensors.

Oudrari, Hassan↗
DOE OSTI2023

Visualization Quality Assessment

Understanding how inaccuracies in visualizations affect users’ perception and understanding of scientific data is hard. Inaccuracies in visualizations are quite common and could arise from a range of sources such as errors in the original dataset arising from compression artifacts, errors in the capturing device, noise during transmission of the data, effects due to the algorithm being used to convert data to visualization images, images generated from neural networks, and sources we have yet to discover. Many image quality assessment metrics have been developed to quantify image errors. However, these are usually focused on “natural images” rather than visualizations of scientific data. Common image quality assessment metrics (IQAs) include MSE, PSNR, perceptual metrics such SSIM, FSIM as well as perceptual metrics using deep learning approaches. However, a critical part of understanding how errors are perceived by humans, and subsequently developing more accurate quality assessment metrics, is through user evaluation studies. The goal of this software is to develop a visualization quality assessment (VQA) process that will enable the generation of VQAs that can be used to quantify errors in scientific data visualizations. The VQA development process will include software to support user evaluation experimental design, analysis of visualization differences against standard quality metrics, and the ability to develop additional VQA metrics specific to scientific visualization images.

Grosset, Andre↗
NASA NTRS2006

A Comparison of Visual Statistics for the Image Enhancement of FORESITE Aerial Images with Those of Major Image Classes

Aerial images from the Follow-On Radar, Enhanced and Synthetic Vision Systems Integration Technology Evaluation (FORESITE) flight tests with the NASA Langley Research Center's research Boeing 757 were acquired during severe haze and haze/mixed clouds visibility conditions. These images were enhanced using the Visual Servo (VS) process that makes use of the Multiscale Retinex. The images were then quantified with visual quality metrics used internally with the VS. One of these metrics, the Visual Contrast Measure, has been computed for hundreds of FORESITE images, and for major classes of imaging--terrestrial (consumer), orbital Earth observations, orbital Mars surface imaging, NOAA aerial photographs, and underwater imaging. The metric quantifies both the degree of visual impairment of the original, un-enhanced images as well as the degree of visibility improvement achieved by the enhancement process. The large aggregate data exhibits trends relating to degree of atmospheric visibility attenuation, and its impact on limits of enhancement performance for the various image classes. Overall results support the idea that in most cases that do not involve extreme reduction in visibility, large gains in visual contrast are routinely achieved by VS processing. Additionally, for very poor visibility imaging, lesser, but still substantial, gains in visual contrast are also routinely achieved. Further, the data suggest that these visual quality metrics can be used as external standalone metrics for establishing performance parameters.

Johnson, Daniel J.↗
NASA NTRS1993

Software metrics: The key to quality software on the NCC project

Network Control Center (NCC) Project metrics are captured during the implementation and testing phases of the NCCDS software development lifecycle. The metrics data collection and reporting function has interfaces with all elements of the NCC project. Close collaboration with all project elements has resulted in the development of a defined and repeatable set of metrics processes. The resulting data are used to plan and monitor release activities on a weekly basis. The use of graphical outputs facilitates the interpretation of progress and status. The successful application of metrics throughout the NCC project has been instrumental in the delivery of quality software. The use of metrics on the NCC Project supports the needs of the technical and managerial staff. This paper describes the project, the functions supported by metrics, the data that are collected and reported, how the data are used, and the improvements in the quality of deliverable software since the metrics processes and products have been in use.

Burns, Patricia J.↗
DOE OSTI2024

Review of Technical Photovoltaic Key Performance Indicators and the Importance of Data Quality Routines

Technical key performance indicators (KPIs) are important metrics used to assess and quantitatively summarize various aspects of photovoltaic (PV) systems, including long-term performance, economic viability, and carbon footprint. Herein, a group of experts of the International Energy Agency's Photovoltaic Power Systems Programme Task 13 collect and describ the most important technical KPIs used in the industry. Thereby, a set of best practices for reliably handling PV system data is presented and the impact of data quality and climatic variability on KPI calculation is investigated. Further, the effective use of technical KPIs allows triggering data-driven and informed decisions to optimize PV systems and providing a comprehensive overview of how PV systems operate across different conditions and climates. With the worldwide growth of the PV industry, more companies operate/own PV systems in different regions, where the climatic and seasonal profiles differ. This requires context-aware evaluation of KPIs, or the judicious application of multiple KPIs, to ensure that each asset is evaluated correctly. Beyond that, there is untapped potential in the utilization of KPIs through geospatial mapping and extrapolation of fleet KPIs. This study demonstrates that the uncertainty in KPI estimation is not well understood and depends on data quality, climatic variability, and system configuration.

14 SOLAR ENERGY↗
DOE OSTI2022

Computational models of direct and indirect X‐ray breast imaging detectors for in silico trials

Abstract Background To facilitate in silico studies that investigate digital mammography (DM) and breast tomosynthesis (DBT), models replicating the variety in imaging performance of the DM and DBT systems, observed across manufacturers are needed. Purpose The main purpose of this work is to develop generic physics models for direct and indirect detector technology used in commercially available systems, with the goal of making them available open source to manufacturers to further tweak and develop the exact in silico replicas of their systems. Methods We recently reported on an in silico version of the SIEMENS Mammomat Inspiration DM/DBT system using an open‐source GPU‐accelerated Monte Carlo x‐ray imaging simulation code (MC‐GPU). We build on the previous version of the MC‐GPU codes to mimic the imaging performances of two other Food and Drug Administration (FDA)‐approved DM/DBT systems, such as Hologic Selenia Dimensions (HSD) and the General Electric Senographe Pristina (GSP) systems. In this work, we developed a hybrid technique to model the optical spread and signal crosstalk observed in the GSP and HSD systems. MC simulations are used to track each x‐ray photon till its first interaction within the x‐ray detector. On the other hand, the signal spread in the x‐ray detectors is modeled using previously developed analytical equations. This approach allows us to preserve the modeling accuracy offered by MC methods in the patient body, while speeding up secondary carrier transport (either electron–hole pairs or optical photons) using analytical equations in the detector. The analytical optical spread model for the indirect detector includes the depth‐dependent spread and collection of optical photons and relies on a pre‐computed set of point response functions that describe the optical spread as a function of depth. To understand the capabilities of the computational x‐ray detector models, we compared image quality metrics like modulation transfer function (MTF), normalized noise power spectrum (NNPS), and detective quantum efficiency (DQE), simulated with our models against measured data. Please note that the purpose of these comparisons with measured data would be to gauge if the model developed as part of this work could replicate commercially used direct and indirect technology in general and not to achieve perfect fits with measured data. Results We found that the simulated image quality metrics such as MTF, NNPS, and DQE were in reasonable agreement with experimental data. To demonstrate the imaging performance of the three DM/DBT systems, we integrated the detector models with the VICTRE pipeline and simulated DM images of a fatty breast model containing a spiculated mass and a calcium oxalate cluster. In general, we found that the images generated using the indirect model appeared more blurred with a different noise texture and contrast as compared to the systems with direct detectors. Conclusions We have presented computational models of three commercially available FDA‐approved DM/DBT systems, which implement both direct and indirect detector technology. The updated versions of the MC‐GPU codes that can be used to replicate three systems are available in open source format through GitHub.

Sengupta, Aunnasha↗
DOE OSTI2021

Quality Control Methods for Advanced Metering Infrastructure Data

While urban-scale building energy modeling is becoming increasingly common, it currently lacks standards, guidelines, or empirical validation against measured data. Empirical validation necessary to enable best practices is becoming increasingly tractable. The growing prevalence of advanced metering infrastructure has led to significant data regarding the energy consumption within individual buildings, but is something utilities and countries are still struggling to analyze and use wisely. In partnership with the Electric Power Board of Chattanooga, Tennessee, a crude OpenStudio/EnergyPlus model of over 178,000 buildings has been created and used to compare simulated energy against actual, 15-min, whole-building electrical consumption of each building. In this study, classifying building type is treated as a use case for quantifying performance associated with smart meter data. This article attempts to provide guidance for working with advanced metering infrastructure for buildings related to: quality control, pathological data classifications, statistical metrics on performance, a methodology for classifying building types, and assess accuracy. Advanced metering infrastructure was used to collect whole-building electricity consumption for 178,333 buildings, define equations for common data issues (missing values, zeros, and spiking), propose a new method for assigning building type, and empirically validate gaps between real buildings and existing prototypes using industry-standard accuracy metrics.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRS2006

A Perspective on DSN System Performance Analysis

This paper discusses the performance analysis effort being carried out in the NASA Deep Space Network. The activity involves root cause analysis of failures and assessment of key performance metrics. The root cause analysis helps pinpoint the true cause of observed problems so that proper correction can be effected. The assessment currently focuses on three aspects: (1) data delivery metrics such as Quantity, Quality, Continuity, and Latency; (2) link-performance metrics such as antenna pointing, system noise temperature, Doppler noise, frequency and time synchronization, wide-area-network loading, link-configuration setup time; and (3) reliability, maintainability, availability metrics. The analysis establishes whether the current system is meeting its specifications and if so, how much margin is available. The findings help identify the weak points in the system and direct attention of programmatic investment for performance improvement.

Deep Space Network (DSN)↗
NASA NTRS2012

Optical Sensors for Planetary Radiant Energy (OSPREy): Calibration and Validation of Current and Next-Generation NASA Missions

A principal objective of the Optical Sensors for Planetary Radiance Energy (OSPREy) activity is to establish an above-water radiometer system as a lower-cost alternative to existing in-water systems for the collection of ground-truth observations. The goal is to be able to make high-quality measurements satisfying the accuracy requirements for the vicarious calibration and algorithm validation of next-generation satellites that make ocean color and atmospheric measurements. This means the measurements will have a documented uncertainty satisfying the established performance metrics for producing climate-quality data records. The OSPREy approach is based on enhancing commercial-off-the-shelf fixed-wavelength and hyperspectral sensors to create hybridspectral instruments with an improved accuracy and spectral resolution, as well as a dynamic range permitting sea, Sun, sky, and Moon observations. Greater spectral diversity in the ultraviolet (UV) will be exploited to separate the living and nonliving components of marine ecosystems; UV bands will also be used to flag and improve atmospheric correction algorithms in the presence of absorbing aerosols. The short-wave infrared (SWIR) is expected to improve atmospheric correction, because the ocean is radiometrically blacker at these wavelengths. This report describes the development of the sensors, including unique capabilities like three-axis polarimetry; the documented uncertainty will be presented in a subsequent report.

Hooker, Stanford B.↗
DOE OSTI2022

Livewire Data Platform-A Solution for Energy Efficient Mobility Systems (EEMS) Data Sharing

This presentation is an overview of progress made on the Livewire Data Platform since June 2021. Livewire is a publicly accessible platform for sharing energy efficiency and mobility research data funded by DOE's Vehicle Technologies Office. Core services and platform capabilities of Livewire include: Free, secure data storage; Access management that allows data owners to control who sees their data; Data collection and preservation; Quality characterization; Detailed access and download metrics; and Increased visibility of projects and data.

energy efficiency mobility↗
DOE OSTI2023

Livewire Data Platform-A Solution for Energy Efficient Mobility Systems (EEMS) Data Sharing

This presentation is an overview of progress made on the Livewire Data Platform since June 2022. Livewire is a publicly accessible platform for sharing energy efficiency and mobility research data funded by DOE's Vehicle Technologies Office. Core services and platform capabilities of Livewire include: Free, secure data storage; Access management that allows data owners to control who sees their data; Data collection and preservation; Quality characterization; Detailed access and download metrics; and Increased visibility of projects and data. Anyone can create an account and access data at https://livewire.energy.gov/.

energy efficiency↗