Search NASA⌕ Search

SEARCH · Search NASA

Results for “data quality”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

A priori checking of the light-response and data quality before extended data collection in pump–probe photocrystallography experiments

In picosecond and slower pump–probe diffraction experiments, collection of response–ratio correlation sets prior to full data collection provides an invaluable confirmation of the existence of a light-induced signal prior to full data collection. If a response to light exposure is observed, the quality of the data being collected can be assessed. We present a number of such correlation plots both for synchrotron and in-house pump–probe data collection.

Coppens, Philip↗
DOE OSTI2026

Machine Learning-Based Anomaly Detection for PMT Data Quality Monitoring in the SBN and DUNE

Maintaining high-quality detector data is essential for achieving the scientific objectives of the Short-Baseline Neutrino (SBN) Program at Fermilab. Current data quality monitoring (DQM) procedures rely primarily on threshold-based metrics and manual inspection of detector monitoring plots, making the detection of subtle or gradually developing anomalies both time-consuming and dependent on expert interpretation. This project developed and evaluated a machine-learning workflow for automatically identifying anomalous photomultiplier tube (PMT) channels in the Short-Baseline Near Detector (SBND) using optical-hit amplitude data. A Python-based analysis program was developed to process ROOT files, extract statistical features describing individual PMT amplitude distributions, and generate feature vectors for anomaly detection. These features were used to train an Isolation Forest model using data representing normal detector operation. The trained model was subsequently applied to independent detector runs to identify channels exhibiting statistically unusual behavior relative to the learned reference response. To support expert interpretation, the workflow generated complementary diagnostic products, including anomaly score distributions, normalized amplitude comparisons, decision-tree visualizations, and principal component analysis (PCA) projections. This project demonstrated the feasibility of integrating unsupervised machine learning into detector data-quality monitoring and developed a complete workflow for automated PMT performance assessment to aid expert-driven review. Beyond its technical contributions, the VFP appointment fostered a research collaboration between Aurora University and Fermilab and provided direct workforce development benefits by training the visiting faculty member in detector-scale machine-learning methods that are now being incorporated into undergraduate coursework and research. The methodology developed here provides a foundation for future applications to ProtoDUNE and other liquid argon time projection chamber (LArTPC) detectors, contributing to ongoing efforts to improve detector reliability, reduce manual monitoring requirements, and enable scalable data quality monitoring for future large-scale neutrino experiments, including the Deep Underground Neutrino Experiment (DUNE).

Colón Santana, Juan A. [Unlisted, US, IL]↗
NASA NTRS2016

International Metadata Standards and Enterprise Data Quality Metadata Systems

Well-documented data quality is critical in situations where scientists and decision-makers need to combine multiple datasets from different disciplines and collection systems to address scientific questions or difficult decisions. Standardized data quality metadata could be very helpful in these situations. Many efforts at developing data quality standards falter because of the diversity of approaches to measuring and reporting data quality. The one size fits all paradigm does not generally work well in this situation. I will describe these and other capabilities of ISO 19157 with examples of how they are being used to describe data quality across the NASA EOS Enterprise and also compare these approaches with other standards.

data quality↗
DOE OSTI2019

Defining a Use Case for ADMS Testbed: Data Quality Requirements for ADMS Deployment

Advanced distribution management systems (ADMS) provide a suite of tools to meet the needs of a modern grid: increased reliability and power quality, improved resiliency and security, reduced costs, and enhanced customer participation. A critical challenge that utilities face with adoption of ADMS is the quality of models and data that the ADMS uses for making control decisions. Data quality has a two-fold impact on ADMS adoption: 1. Data quality improvement might constitute up to 25% of ADMS deployment costs. 2. The accuracy of data and models used by the ADMS affects the utility's ability to meet its operational objectives. Thus, quantifying the data quality requirements and its impact on performance is critical to reducing the overall cost of deployment, enabling increased adoption and ensuring that the ADMS performs as specified. This paper offers the motivation, methodology, and evaluation strategy to fill this critical gap.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2019

Provenance–aware workflow for data quality management and improvement for large continuous scientific data streams

Data quality assessment, management and improvement is integral part of any big data intensive scientific research to ensure accurate, reliable and reproducible science discoveries. The task of maintaining the quality of data, however, is non-trivial and pose a challenge for a program like Department of Energy’s Atmospheric Radiation Measurement (ARM) that collects data from hundreds of instruments across the world,and distributes thousands of streaming data products that are continuously growing in near-real-time in an archive 1.7 Petabyte in size and growing. We present a computational data processing workflow to collect the data quality issue via an easy and intuitive web-based portal that allows reporting of any quality issues for any site, facility or instruments at a granularity down to individual variable in the data files. Portal allows instrument specialists and scientists to provide corrective actions in form of symbolic equation. A parallel processing framework applies the data improvement to large volume of data in efficient parallel environment, while optimizing data transfer and file I/O operations. Corrected files are systematically versioned and archived. A provenance tracking module tracks and records any change made to the data during its entire life cycle which are communicated transparently to the scientific users. Developed in Python using open source technologies, software architecture enables efficient and fast management and improvement of data in an operational data center environment.

Kumar, Jitendra↗
DOE OSTI2019

MY EXES and EDM for Automated Remediation Data Quality Assessments - 19053

MY Electronic data exchange and Evaluation System (EXES) is a new web-based technology providing external admin access and full control to the Web-based EXES tool developed by GD for EPA for automated environmental remediation data quality assessments. EXES replaces most manual assessment processes, reducing remediation data review time and costs and ensuring that data are of known and documented quality. This tool is available from EPA to federal agencies and remediation firms and has successfully been used to support over 190 site investigations, 105 Remedial investigations and over 20 compliance monitoring efforts. The new MY EXES functionality has been developed for project managers and decision makers to setup projects, track receipt of environmental remediation monitoring data, assess data quality, and manage data. It is designed to be flexible and configurable to meet any government agency's data review and data management needs. It is accessible 24/7 through a secure portal. MY EXES efficiently evaluates analytical data regardless of the underlying method and data reporting requirements. It is a centralized project and data assessment tool that is scalable, flexible and configurable; improving data quality while reducing cost through the following: - Provides users with the ability to create a project and to specify the analytical methods corresponding to the data that will be submitted for that project. A project can be modified and managed anytime during the project's life cycle. - Allows users to manage a library of analytical methods that can be made available for association to their projects. - Users can specify the automated tests and assessment that needs to be performed on the analytical data deliverables based on specific requirements in the project's Quality Assurance Project Plan (QAPP). - Users specify: - The format of the incoming data deliverables from the laboratory; - The type and level of review services to be performed; - The assessment deliverables to be generated - output spreadsheets and reports; - Project owner sets up system access and permissions for members of the project team. - A faster (600%) and less expensive (85%) process than the traditional data usability/validation. The EXES Data Manager application (EDM) provides a centralized data repository to view and retrieve generated reports, spreadsheets, and other data deliverables as defined by the client, eliminating the need for data transfer between project team members. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2025

Data Quality Assessment Process for Real-Time Data-Driven Traffic Microsimulation of Smart Corridor

Smart corridor digital twins are often created for the development and evaluation of emerging intelligent transportation systems and Connected and Autonomous Vehicle (CAV) technologies. However, limited guidance exists for data quality assessment for digital twin development. To address this, this paper discusses the data quality assessment utilized to develop data-driven real-time microscopic simulation models, i.e., digital twins, for two separate smart corridors: the North Avenue Smart Corridor in Atlanta, GA, and the Martin Luther King Smart Corridor in Chattanooga, Tennessee. This paper provides a summary of the author’s investigations of data requirements and data characteristics for the given smart corridor digital twin development efforts. With a focus on data, this summary includes a description of the data investigation process, key data issues observed, and strategies to address observed issues. Discussion is provided to help expand the lessons from these studies to other digital twin development efforts.

Saroj, Abhilasha [ORNL] (ORCID:0000000191178063)↗
DOE OSTI2018

Spring Boot based REST API to Improve Data Quality Report Generation for Big Scientific Data: ARM Data Center Example

Web application technologies are growing rapidly with continuous innovation and improvements. This paper focuses on the popular Spring Boot [1] java-based framework for building web and enterprise applications and how it provides the flexibility for service-oriented architecture (SOA). One challenge with any Spring-based applications is its level of complexity with configurations. Spring Boot makes it easy to create and deploy stand-alone, production-grade Spring applications with very little Spring configuration. Example, if we consider Spring Model-View-Controller (MVC) framework [2], we need to configure dispatcher servlet, web jars, a view resolver, and component scan among other things. To solve this, Spring Boot provides several Auto Configuration options to setup the application with any needed dependencies. Another challenge is to identify the framework dependencies and associated library versions required to develop a web application. Spring Boot offers simpler dependency management by using a comprehensive, but flexible, framework and the associated libraries in one single dependency, which provides all the Spring related technology that you need for starter projects as compared to CRUD web applications. This framework provides a range of additional features that are common across many projects such as embedded server, security, metrics, health checks, and externalized configuration. Web applications are generally packaged as war and deployed to a web server, but Spring Boot application can be packaged either as war or jar file, which allows to run the application without the need to install and/or configure on the application server. In this paper, we discuss how Atmospheric Radiation Measurement (ARM) Data Center (ADC) at Oak Ridge National Laboratory, is using Spring Boot to create a SOA based REST [4] service API, that bridges the gap between frontend user interfaces and backend database. Using this REST service API, ARM scientists are now able to submit reports via a user form or a command line interface, which captures the same data quality or other important information about ARM data.

Guntupally, Kavya↗
DOE OSTI2026

Livewire: A Model Platform for Data Quality Assessment and AI Readiness Across DOE Missions

High-quality, well-governed data is essential for accelerating discovery and achieving operational excellence across DOE and national laboratory missions. The Livewire Data Platform is a DOE-supported platform that offers automated assessments of data quality, standardization, provenance, and Artificial Intelligence (AI) readiness. It allows researchers and data practitioners to systematically and easily evaluate datasets against established governance criteria and prepare them for advanced analytics. Livewire addresses critical challenges in DOE's data ecosystem with integrated capabilities for metadata validation, provenance tracking, and schema alignment. This platform's automated workflows assist users in identifying data quality gaps, enhancing interoperability between datasets collected from various stakeholders, and ensuring compliance with DOE data standards, all while reducing manual curation efforts. Additionally, we will discuss its AI readiness framework, which is being developed to prepare datasets for training models, developing advanced analytic tools, and machine learning applications. Using some of the more than one hundred tabular datasets on Livewire, processed with this open-source methodology, we will demonstrate how Livewire can serve as a model for scalable, standards-driven data management. This approach provides a pathway to leverage existing and future datasets within the DOE, boosting innovation and efficiency across national laboratories.

33 - ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2021

Pycheron: A Python-Based Seismic Waveform Data Quality Control Software Package

Supplementing an existing high-quality seismic monitoring network with openly available station data could improve coverage and decrease magnitudes of completeness; however, this can present challenges when varying levels of data quality exist. Without discerning the quality of openly available data, using it poses significant data management, analysis, and interpretation issues. Incorporating additional stations without properly identifying and mitigating data quality problems can degrade overall monitoring capability. If openly available stations are to be used routinely, a robust, automated data quality assessment for a wide range of quality control (QC) issues is essential. To meet this need, we developed Pycheron, a Python-based library for QC of seismic waveform data. Pycheron was initially based on the Incorporated Research Institutions for Seismology’s Modular Utility for STAtistical kNowledge Gathering but has been expanded to include more functionality. Pycheron can be implemented at the beginning of a data processing pipeline or can process stand-alone data sets. Its objectives are to (1) identify specific QC issues; (2) automatically assess data quality and instrumentation health; (3) serve as a basic service that all data processing builds on by alerting downstream processing algorithms to any quality degradation; and (4) improve our ability to process orders of magnitudes more data through performance optimizations. This article provides an overview of Pycheron, its features, basic workflow, and an example application using a synthetic QC data set.

58 GEOSCIENCES↗
NASA NTRS1974

Systematic monitoring and evaluation of M7 scanner performance and data quality

An investigation was conducted to provide the information required to maintain data quality of the Michigan M7 Multispectral scanner by systematic checks on specific system performance characteristics. Data processing techniques which use calibration data gathered routinely every mission have been developed to assess current data quality. Significant changes from past data quality are thus identified and attempts made to discover their causes. Procedures for systematic monitoring of scanner data quality are discussed. In the solar reflective region, calculations of Noise Equivalent Change in Radiance on a permission basis are compared to theoretical tape-recorder limits to provide an estimate of overall scanner performance. M7 signal/noise characteristics are examined.

Stewart, S.↗
NASA NTRS1984

Preliminary Evaluation of Thematic Mapper Image Data Quality

Thematic Mapper (TM) data from Mississippi County, Arkansas, and Webster County, Iowa, were examined for the purpose of evaluating the image data quality of the TM which was launched on board the LANDSAT-4 spacecraft. Preliminary clustering and principal component analysis indicates that the middle infrared and thermal infrared data of TM appear to add significant information over that of the near IR and visible bands of the multispectral scanner data. Moreover, the higher spatial resolution of TM appears to provide better definition of the edges and the within variability of agricultural fields. The geometric performance of TM data, without ground control correction, was found to exceed expectations. The modulation transfer function for the 1.65 m band was found to agree with prelaunch specifications when the effects of the GSFC cubic convolution and the atmosphere were removed. The band to band registration for the bands within the noncooled focal plane was found to be better than specified. However, the middle infrared and thermal infrared, which are on a separate cooled focal plane were found to be misregistered and were significantly worse than prelaunch specifications.

Macdonald, R. B.↗
DOE OSTI2021

AI Enabled Data Quality Monitoring with Hydra

Data quality monitoring is critical to all experiments impacting the quality of any physics results. Traditionally, this is done through an alarm system, which detects low level faults, leaving higher level monitoring to human crews. Artificial Intelligence is beginning to find its way into scientific applications, but comes with difficulties, relying on the acquisition of new skill sets, either through education or acquisition, in data science. This paper will discuss the development and deployment of the Hydra monitoring system in production at Gluex. It will show how “off-the-shelf” technologies can be rapidly developed, as well as discuss what sociological hurdles must be overcome to successfully deploy such a system. Early results from production running of Hydra will also be shared as well as a future outlook for development of Hydra.

Britton, Thomas↗
DOE OSTI2014

Comparison of Data Quality of NOAA's ISIS and SURFRAD Networks to NREL's SRRL-BMS

This report provides analyses of broadband solar radiometric data quality for the National Oceanic and Atmospheric Administration's Integrated Surface Irradiance Study and Surface Radiation Budget Network (SURFRAD) solar measurement networks. The data quality of these networks is compared to that of the National Renewable Energy Laboratory's Solar Radiation Research Laboratory Baseline Measurement System (SRRL-BMS) native data resolutions and hourly averages of the data from the years 2002 through 2013. This report describes the solar radiometric data quality testing and flagging procedures and the method used to determine and tabulate data quality statistics. Monthly data quality statistics for each network were plotted by year against the statistics for the SRRL-BMS. Some of the plots are presented in the body of the report, but most are in the appendix. These plots indicate that the overall solar radiometric data quality of the SURFRAD network is superior to that of the Integrated Surface Irradiance Study network and can be comparable to SRRL-BMS.

14 SOLAR ENERGY↗
DOE OSTI2024

Hydra: Computer Vision for Data Quality Monitoring

Hydra: Computer Vision for Data Quality Monitoring Mar 13, 2024, 4:15 PM 30m Charles B. Wang Center, Stony Brook University Poster Track 1: Computing Technology for Physics Research Poster session with coffee break Speaker Ayan Roy Description Hydra is an extensible framework for training, managing, and deploying machine learning models for near real-time data quality monitoring. It is designed to take some of the burden off of shift crews by providing ‘round-the-clock’ monitoring of plots representing the data being collected. The Hydra system is backed by a database which is leveraged for near push button training and is primarily controlled and viewed through a set of web interfaces. This web interface contains a simple-to-use web based GUI for labeling of the datasets used in training; making it possible to label thousands of images quickly and efficiently. To aid in the analysis of Hydra inferences gradCAM, a method of interpretability, is performed and overlaid on the target image, highlighting regions of interest, making diagnosis of problems much faster. Development began in 2019 for the GlueX Experiment in Hall-D, the Hydra system has grown to encompass all of the experimental halls at Jefferson Laboratory. This talk will focus on the features of Hydra as well as provide details of the challenges present with deploying to disparate experimental halls. With a roadmap of development established, Hydra aims to continue to grow in richness of feature set and expand to encompass other monitoring tasks in differing communities.

Roy, Ayan↗
NASA NTRS1984

Evaluation of Landsat-4 Thematic Mapper and multispectral scanner data quality

Landsat-4 image data quality was evaluated for test sites in Iowa and Illinois. Radiometric and geometric quality was tested and an applications evaluation was carried out using a cooling-pond thermal-mapping example. Geometric quality was found to be generally very good. Small errors were found in registration of the middle IR bands of the TM and the thermal IR band was found to be misregistered by one 120-meter pixel. Radiometric quality of the TM is excellent with only minor striping effects.

Bartolucci, L. A.↗