Search NASA⌕ Search

SEARCH · Search NASA

Results for “data quality issues”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Genomes to Proteomes

Biologists are awash with genomic sequence data. In large part, this is due to the rapid acceleration in the generation of DNA sequence that occurred as public and private research institutes raced to sequence the human genome. In parallel with the large human genome effort, mostly smaller genomes of other important model organisms were sequenced. Projects following on these initial efforts have made use of technological advances and the DNA sequencing infrastructure that was built for the human and other organism genome projects. As a result, the genome sequences of many organisms are available in high quality draft form. While in many ways this is good news, there are limitations to the biological insights that can be gleaned from DNA sequences alone; genome sequences offer only a bird's eye view of the biological processes endemic to an organism or community. Fortunately, the genome sequences now being produced at such a high rate can serve as the foundation for other global experimental platforms such as proteomics. Proteomic methods offer a snapshot of the proteins present at a point in time for a given biological sample. Current global proteomics methods combine enzymatic digestion, separations, mass spectrometry and database searching for peptide identification. One key aspect of proteomics is the prediction of peptide sequences from mass spectrometry data. Global proteomic analysis uses computational matching of experimental mass spectra with predicted spectra based on databases of gene models that are often generated computationally. Thus, the quality of gene models predicted from a genome sequence is crucial in the generation of high quality peptide identifications. Once peptides are identified they can be assigned to their parent protein. Proteins identified as expressed in a given experiment are most useful when compared to other expressed proteins in a larger biological context or biochemical pathway. In this chapter we will discuss the automatic annotation and the generation of high quality gene models, the setup and execution of global proteomics experiments that are quantitative and statistically rigorous and finally add biological context to proteomics.

59 BASIC BIOLOGICAL SCIENCES↗

The AmeriFlux Data Activity and Data System: An Evolving Collection of Data Management Techniques, Tools, Products and Services

The Carbon Dioxide Information Analysis Center (CDIAC) at Oak Ridge National Laboratory (ORNL), USA has provided scientific data management support for the U.S. Department of Energy and international climate change science since 1982. Over this period, climate change science has expanded from research focusing on basic understanding of geochemical cycles, particularly the carbon cycle, to integrated research addressing climate change impacts, vulnerability, adaptation, and mitigation. Interests in climate change data and information worldwide have grown remarkably and, as a result, so have demands and expectations for CDIAC’s data systems. To meet the growing demands, CDIAC’s strategy has been to design flexible data systems using proven technologies blended with new, evolving technologies and standards. CDIAC development teams are multidisciplinary and include computer science and information technology expertise, but also scientific expertise necessary to address data quality and documentation issues and to identify data products and system capabilities needed by climate change scientists. CDIAC has learned there is rarely a single commercial tool or product readily available to satisfy long-term scientific data system requirements (i.e., one size does not fit all and the breadth and diversity of environmental data are often too complex for easy use with commercial products) and typically deploys a variety of tools and data products in an effort to provide credible data freely to users worldwide. Like many scientific data management applications, CDIAC’s data systems are highly customized to satisfy specific scientific usage requirements (e.g., developing data products specific for model use) but are also designed to be flexible and interoperable to take advantage of new software engineering techniques, standards (e.g., metadata standards) and tools and to support future Earth system data efforts (e.g., ocean acidification). CDIAC has provided data management support for numerous long-term measurement projects crucial to climate change science. One current example is the AmeriFlux measurement network. AmeriFlux provides continuous measurements from forests, grasslands, wetlands, and croplands in North, Central, and South America and offers important insight about carbon cycling in terrestrial ecosystems. We share our approaches in satisfying the challenges of delivering AmeriFlux data worldwide to benefit others with similar challenges handling climate change data, further heighten awareness and use of an outstanding ecological data resource, and highlight expanded software engineering applications being used for climate change measurement data.

Boden, Thomas A.↗

Ambiguity of Quality in Remote Sensing Data

This slide presentation reviews some of the issues in quality of remote sensing data. Data "quality" is used in several different contexts in remote sensing data, with quite different meanings. At the pixel level, quality typically refers to a quality control process exercised by the processing algorithm, not an explicit declaration of accuracy or precision. File level quality is usually a statistical summary of the pixel-level quality but is of doubtful use for scenes covering large areal extents. Quality at the dataset or product level, on the other hand, usually refers to how accurately the dataset is believed to represent the physical quantities it purports to measure. This assessment often bears but an indirect relationship at best to pixel level quality. In addition to ambiguity at different levels of granularity, ambiguity is endemic within levels. Pixel-level quality terms vary widely, as do recommendations for use of these flags. At the dataset/product level, quality for low-resolution gridded products is often extrapolated from validation campaigns using high spatial resolution swath data, a suspect practice at best. Making use of quality at all levels is complicated by the dependence on application needs. We will present examples of the various meanings of quality in remote sensing data and possible ways forward toward a more unified and usable quality framework.

Lynnes, Christopher↗

FAST FY 2018 Federal Fleet Data Call: Looking at Data Quality

This presentation provides an overview of fiscal year 2018 federal motor vehicle fleet data at the individual vehicle level during the fall of 2018, a preliminary summary of the data collected, how the the collecting project has reviewed that information for potential quality issues, and an overview of two key areas within the dataset that represent data quality issues. This presentation will be given at the April 2019 FedFleet training event, hosted by the US General Services Administration in Washington, DC. The information is collected through the Federal Automotive Statistical Tool (FAST) project. FAST is a Web-based information system managed by the US Department of Energy, the US General Services Administration, and the Energy Information Administration. FAST is used to collect information about the fleet of motor vehicles used and managed by the Federal government. FAST is developed and maintained by DOE's Idaho National Laboratory (INL).

99 GENERAL AND MISCELLANEOUS↗

Final Draft of RACER Audit

This document describes the approach Waste and Environmental Services - Environmental Data and Analysis plans to take to resolve the issues presented in a recent audit of the WES-EDA Environmental Database relative to the RACER database. A majority of the issues discovered in the audit will be resolved in May 2011 when the WES-EDA Environmental Database, along with other LANL databases, are integrated and moved to a new vendor providing an Environmental Information Management (EIM) system that allows reporting capabilities for all users directly from the database. The EIM system will reside in a publicly accessible LANL cloud-based software system. When this transition occurs, the data quality, completeness, and access will change significantly. In the remainder of this document, this new structure will be referred to as the LANL Cloud System In general, our plan is to address the issues brought up in this audit in three ways: (1) Data quality issues such as units and detection status, which impinge upon data usability, will be resolved as soon possible so that data quality is maintained. (2) Issues requiring data cleanup, such as look up tables, legacy data, locations, codes, and significant data discrepancies, will be addressed as resources permit. (3) Issues associated with data feed problems will be eliminated by the LANL Cloud System, because there will be no data feed. As discussed in the paragraph above, in the future the data will reside in a publicly accessible system. Note that report writers may choose to convert, adapt, or simplify the information they receive officially through our data base, thereby introducing data discrepancies between the data base and the public report. It is not always possible to incorporate and/or correct these errors when they occur. Issues in the audit will be discussed in the order in which they are presented in the audit report. Clarifications will also be noted as the audit report was a draft document, at the time of this response.

AUDITS↗

Quality Control Methods of Tower Data at Kennedy Space Center’s Launch Complex 39-B and the USAF Cape Canaveral 500 Foot Tower

The National Aeronautics and Space Administration (NASA) has long used meteorological data from weather towers located at Kennedy Space Center (KSC) and the United States Air Force’s Eastern Range(ER) in support of their various launch vehicles and climate studies. Some of the most valuable data is gathered at the Launch Complex 39-B (LC39-B) and the 500 foot tower located approximately two miles west of LC39-B and three miles north of the Vehicle Assembly Building (VAB). The data from LC39-B gives NASA Space Launch System (SLS) engineers valuable insight into the weather the vehicle can expect at the pad, while the 500 foot tower has a longer period of record and has been used by the Shuttle and SLS programs. However, numerous data quality (DQ) control issues have arisen when utilizing this data for analysis. In addition, instrumentation tends to deteriorate faster than normal due to the corrosive nature of the high salt content in the air. NASA’s Natural Environments branch has developed several QC databases of these towers, but these studies were for fixed periods of records. EV44 identified a need for a continually updated and QC’ed database of tower data not only to provide along term QC’ed database for vehicle and climate analyses, but also to provide the capability to investigate recent weather events such as downbursts and other high wind events. This study follows the methods developed in prior Natural Environment QC tower databases, which includes variable specific QC thresholds and checks used to generate QC flags. However, this study developed new techniques such as the development and design of the graphical user interface (GUI) for manual verification, and the creation of the final monthly QC files.

Quality Control↗

Pycheron: A Python-Based Seismic Waveform Data Quality Control Software Package

Supplementing an existing high-quality seismic monitoring network with openly available station data could improve coverage and decrease magnitudes of completeness; however, this can present challenges when varying levels of data quality exist. Without discerning the quality of openly available data, using it poses significant data management, analysis, and interpretation issues. Incorporating additional stations without properly identifying and mitigating data quality problems can degrade overall monitoring capability. If openly available stations are to be used routinely, a robust, automated data quality assessment for a wide range of quality control (QC) issues is essential. To meet this need, we developed Pycheron, a Python-based library for QC of seismic waveform data. Pycheron was initially based on the Incorporated Research Institutions for Seismology’s Modular Utility for STAtistical kNowledge Gathering but has been expanded to include more functionality. Pycheron can be implemented at the beginning of a data processing pipeline or can process stand-alone data sets. Its objectives are to (1) identify specific QC issues; (2) automatically assess data quality and instrumentation health; (3) serve as a basic service that all data processing builds on by alerting downstream processing algorithms to any quality degradation; and (4) improve our ability to process orders of magnitudes more data through performance optimizations. This article provides an overview of Pycheron, its features, basic workflow, and an example application using a synthetic QC data set.

58 GEOSCIENCES↗

Physics-Informed Sparse Gaussian Process for Probabilistic Stability Analysis of Large-Scale Power System with Dynamic PVs and Loads

This work proposes a physics-informed sparse Gaussian process (SGP) for probabilistic stability assessment of large-scale power systems in the presence of uncertain dynamic PVs and loads. The differential and algebraic equations considering uncertainties from dynamic PVs and loads are reformulated to a nonlinear mapping relationship that allows the application of SGP. Thanks to the nonparametric characteristic of Gaussian process, the proposed framework does not require distributions of uncertain inputs and this distinguishes it from existing approaches. As the original Gaussian process is not scalable to large-scale systems with high dimensional uncertain inputs, this paper develops the SGP with a stochastic variational inference technique. It leads to approximately two orders of complex reduction. A data pre-processing step is also introduced to tackle the coexistence of stable and unstable cases by sample clustering and constructing separate SGPs. The probabilistic transient stability index is analyzed to assess system stability under different uncertain dynamics loads and PVs. Comparisons are performed with the sampling-based, the polynomial chaos expansion-based, and traditional Gaussian process-based methods on the modified IEEE 118-bus and Texas 2000-bus systems under various scenarios, including different levels of uncertainties and the existence of nonlinear correlations among dynamic PVs. The impacts of data quality and quantity issues are also investigated. It is shown that the proposed SGP achieves significantly improved computational efficiency while maintaining high accuracy with a limited number of data.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Regulatory Testing of WTP HL W Glasses for Compliance with Delisting Requirements, VSL-03R3780-1, Rev. 1

(Part of the data collected for this work and discussed in this report was subject to data quality and bias issues. All the affected tests have subsequently been repeated, as directed by the Waste Treatment Plant Project, and new statistical analyses have been performed on the revised data set. A subsequently-issued report supercedes this report and describes the revised data, together with the revised composition-property models (Kot et al. 2004). The reader should refer to the new report for discussion of the revised data set and composition-property (TCLP cadmium release) models. None of the data for the spike glasses, designed for Case 1 and Case 2 COPCs testing, were affected by this issue.)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

High-Throughput Method for Strontium Isotope Analysis by Multi-Collector-Inductively Coupled Plasma-Mass Spectrometer

This technical report presents the details of the Sr column configuration and the high-throughput Sr separation protocol. Data showing the performance of the method as well as the best practices for optimizing Sr isotope analysis by MC-ICP-MS is presented. Lastly, this report offers tools for data handling and data reduction of Sr isotope results from the Thermo Scientific Neptune software to assist in data quality assurance, which help avoid issues of data glut associated with high sample throughput rapid analysis.

47 OTHER INSTRUMENTATION↗

High-Throughput Method for Strontium Isotope Analysis by Multi-Collector-Inductively Coupled Plasma-Mass Spectrometer

This technical report presents the details of the Sr column configuration and the high-throughput Sr separation protocol. Data showing the performance of the method as well as the best practices for optimizing Sr isotope analysis by MC-ICP-MS is presented. Lastly, this report offers tools for data handling and data reduction of Sr isotope results from the Thermo Scientific Neptune software to assist in data quality assurance, which help avoid issues of data glut associated with high sample throughput rapid analysis.

47 OTHER INSTRUMENTATION↗

Sensor enabled data-driven predictive analytics for modeling and control with high penetration of DERs in distribution systems

The electric power grid is undergoing a tremendous transformation due to the increasing penetration of renewable energy resources beginning with wind and more recently with the distributed energy resources (DERs) such as solar and battery storage. DERs have dramatically changed the role of the distribution systems in the overall power grid, and they are expected to contribute a significant portion of power generation in the future. If current trends for DERs continue, system operation and control will need to change dramatically for improved grid reliability and resiliency. As renewable resources increase in penetration, new and challenging operational, planning, and design problems are expected to emerge. Some of the key challenges that arise in the planning and operation of the future grid are: 1) Quantifying the impact of high DER penetration in distribution systems on bulk grid behavior over multiple time scales. 2) Identifying whether a particular DER configuration/settings have a large impact on the overall grid behavior. These challenges can be addressed in an offline manner using detailed T&D grid models and they can also be addressed in an online manner using sensor measurements. In particular, the advancement and planned growth in sensor technology in power grid over various voltage levels provide us with a unique opportunity to tackle these challenges from a data analytic perspective without needing detailed T&D grid models. A few questions that naturally arise when addressing the challenges from DERs using sensor data are: 1) How can we use limited sensor measurements to monitor & control voltage stability and small signal stability of the bulk system? 2) How can we ensure that the developed data analytic methods are robust to data availability and quality issues? 3) How can we compute the developed analytics in a scalable manner using streaming measurements? In this project, we addressed the aforementioned challenges arising from DERs and answered the questions raised above on how to effectively use the sensor measurements to enhance the reliability and performance of the electric grid. Thus, the overarching goal of this project is to develop effective reduced/representative system models from data that make the computational complexity sufficiently manageable so as to be useful to simulate, analyze, and even control complex non-linear power systems dynamics with large penetrations of DERs. In order to achieve the objective, the project team established a four-fold technical approach 1) Formulated a combined transmission-distribution co-simulation framework for data generation and validation, 2) Derived reduced/representative models of power systems based on data-driven methods for efficient computation and appropriate representation of system behavior, 3) Developed data driven characterization of power system behavior based on transfer operator theory, machine learning and optimization for model estimation, 4) Incorporated a scalable data management and processing architecture using distributed Kafka streaming applications that coordinate input data streams to the developed data analytics. The key accomplishments of the project are: 1) Development of a scalable multi-timescale T&D co-simulation framework (both for steady state and for dynamic co-simulation) using commercial solvers (PSSE and GridLAB-D). The steady-state T&D co-simulation interface is shared with our industry partner (PJM). 2) A structured reduced order dynamic model of distribution systems that can represent partial motor stalling along with a systematic procedure to derive the model parameters. 3) A PMU based online method to monitor, localize and mitigate fault-induced delayed voltage recovery using DER reactive support and load control in distribution systems. 4) Development of linear operator based robust methodologies for dynamic state estimation, uncertainty quantification, system identification and trajectory prediction for power system dynamics. 5) An adaptive damping control for utilizing wind energy resources to provide oscillation damping and system stability. 6) Implementation of Kafka-based framework for efficient processing of streaming data using Linux-based local virtual environment.

DER integration↗

Leveraging large language models to address data scarcity in machine learning for graphene synthesis

Machine learning in experimental materials science faces significant challenges due to the scarcity of data, which are costly and time-consuming to generate, particularly when relying on in-house experiments. Literature data mining offers a potential solution but introduces issues like mixed data quality, inconsistent formats, and non-uniform reporting of synthesis parameters, resulting in partially missing and heterogeneous features across the dataset. Here, we propose data imputation and feature engineering methods that employ pre-trained large language models (LLMs) to enhance machine learning performance on scarce, heterogeneous datasets, demonstrated on graphene CVD synthesis data and the ML-HydPARK hydrogen storage dataset. GPT models perform data imputation via tailored prompting and semantic normalization of inconsistently reported features through embeddings, for example, to harmonize the complex nomenclature of CVD substrates. Beyond yielding more diverse and richer feature representations than traditional methods such as K-nearest neighbors (KNN) and Multivariate Imputation by Chained Equations (MICE), LLM-based data imputation is evaluated against dataset characteristics and prompting strategies. We vary the level of autonomy granted to the LLM, from generic prompting that leverages pre-trained knowledge for autonomous data generation to data-informed prompting that constrains outputs using target-specific information, and demonstrate which level of autonomy yields superior imputation performance across datasets and feature types. The proposed data engineering methods markedly improve downstream performance; for example, in graphene layer number classification using a support vector machine (SVM), binary accuracy increases from 39% to 65% and ternary accuracy from 52% to 72%. Fine-tuning experiments on both datasets show that combining our proposed LLM-based data imputation and feature encoding methods with numerical machine learning predictors outperforms standalone fine-tuned LLM predictors in data-scarce settings. The proposed strategies emphasize data enhancement techniques rather than refining learning architectures or regularizing loss functions, offering a broadly applicable framework for improving machine learning performance on scarce, inhomogeneous datasets.

Chemical vapor deposition↗

Rainfall and Snowfall Observations by the Airborne Dual-frequency Precipitation Radar during the Wakasa Bay Experiment

Radar data obtained through the NASA/JPL Airborne Precipitation Radar APR-2 during the Wakasa Bay Experiment in January/February 2003 were processed to obtain calibrated reflectivity measurements, rainfall/snowfall velocity measurements, classification of the surface type and detection of the boundaries of the melting layer of precipitation. In this paper the processing approach is described together with an overview of the resulting data quality and known issues.

field campaign↗

Comparison of building energy use data between the United States and China

Buildings in the United States and China consumed 41percent and 28percent of the total primary energy in 2011, respectively. Good energy data are the cornerstone to understanding building energy performance and supporting research, design, operation, and policy making for low energy buildings. This paper presents initial outcomes from a joint research project under the U.S.-China Clean Energy Research Center for Building Energy Efficiency. The goal is to decode the driving forces behind the discrepancy of building energy use between the two countries; identify gaps and deficiencies of current building energy monitoring, data collection, and analysis; and create knowledge and tools to collect and analyze good building energy data to provide valuable and actionable information for key stakeholders. This paper first reviews and compares several popular existing building energy monitoring systems in both countries. Next a standard energy data model is presented. A detailed, measured building energy data comparison was conducted for a few office buildings in both countries. Finally issues of data collection, quality, sharing, and analysis methods are discussed. It was found that buildings in both countries performed very differently, had potential for deep energy retrofit, but that different efficiency measures should apply.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Sci-Fri AM: Quality, Safety, and Professional Issues 02: Recent work on TQC Suite and Data from a National survey on Community Uptake

The Canadian Partnership for Quality Radiotherapy (CPQR) and the Canadian Organization of Medical Physicist’s (COMP) Quality Assurance and Radiation Safety Advisory Committee (QARSAC) have worked together in the development of a suite of Technical Quality Control (TQC) Guidelines for radiation treatment equipment and technologies, that outline specific performance objectives and criteria that equipment should meet in order to assure an acceptable level of radiation treatment quality. Early community engagement and uptake survey data showed 70% of Canadian centers are part of this process and that the data in the guideline documents reflect, and are influencing the way Canadian radiation treatment centres run their technical quality control programs. As the TQC development framework matured as a cross-country initiative, guidance documents have been developed in many clinical technologies. Recently, there have been new TQC documents initiated for Gamma Knife and Cyberknife technologies where the entire communities within Canada are involved in the review process. At the same time, QARSAC reviewed the suite as a whole for the first time and it was found that some tests and tolerances overlapped across multiple documents as single tests could pertain to multiple quality control areas. The work to streamline the entire suite has allowed for improved usability of the suite while keeping the integrity of single quality control areas. The suite will be published by the JACMP, in the coming year.

60 APPLIED LIFE SCIENCES↗

About a method for compressing X-ray computed microtomography data

The management of scientific data is of high importance especially for experimental techniques that produce big data volumes. Such a technique is x-ray computed tomography (CT) and its community has introduced advanced data formats which allow for better management of experimental data. Rather than the organization of the data and the associated meta-data, the main topic on this work is data compression and its applicability to experimental data collected from a synchrotron-based CT beamline at the Elettra-Sincrotrone Trieste facility (Italy) and studies images acquired from various types of samples. This study covers parallel beam geometry, but it could be easily extended to a cone-beam one. The reconstruction workflow used is the one currently in operation at the beamline. Contrary to standard image compression studies, this manuscript proposes a systematic framework and workflow for the critical examination of different compression techniques and does so by applying it to experimental data. Beyond the methodology framework, this study presents and examines the use of JPEG-XR in combination with HDF5 and TIFF formats providing insights and strategies on data compression and image quality issues that can be used and implemented at other synchrotron facilities and laboratory systems. In conclusion, projection data compression using JPEG-XR appears as a promising, efficient method to reduce data file size and thus to facilitate data handling and image reconstruction

data compression↗

Elucidating and predicting the dynamic evolution of water and land systems due to natural and energy-related forcings

Focal Area(s): 3. Insight gleaned from complex data (both observed and simulated) using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge-guided AI; & 1. Data acquisition and assimilation enabled by machine learning, AI, and advanced methods including experimental/network design/optimization, unsupervised learning (including deep learning), and hardware-related efforts involving AI (e.g., edge computing). Science Challenge: Interactions between water, land, and energy systems are complex and occur on a variety of scales, ranging from local to basinal to regional. Accurately predicting the behavior of ground water and surface water systems for 5-10 years and beyond requires an understanding of the current system and the ability to model both the natural system at scale and human-induced forcings related to energy and other activities. Artificial intelligence and machine learning (AI/ML) combined with modern compilation and integration efforts for U.S. groundwater and surface water systems present potential solutions to bolstering detailed physics-based models of these systems. Big data tied with ML and physics-based modeling can drive breakthroughs in understanding the earth system, but research is often impeded by data access (e.g., privacy issues), quality, formats, gaps, multi-source, multi-scale, integration, and spatiotemporal challenges. Effective integration of real data and simulated (synthetic) data that fill gaps is critical. Overcoming these complex data and model integration challenges will enable a transformational approach to acquiring enhanced understanding of environmental systems.

54 ENVIRONMENTAL SCIENCES↗