Search NASA⌕ Search

SEARCH · Search NASA

Results for “data integration.”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2015

Data integration reveals key homeostatic mechanisms following low dose radiation exposure

The goal of this study was to define pathways regulated by low dose radiation to understand how biological systems respond to subtle perturbations in their environment and prioritize pathways for human health assessment. Using an in vitro 3-D human full thickness skin model, we have examined the temporal response of dermal and epidermal layers to 10 cGy X-ray using transcriptomic, proteomic, phosphoproteomic and metabolomic platforms. Bioinformatics analysis of each dataset independently revealed potential signaling mechanisms affected by low dose radiation, and integrating data shed additional insight into the mechanisms regulating low dose responses in human tissue. We examined direct interactions among datasets (top down approach) and defined several hubs as significant regulators, including transcription factors (YY1, MYC and CREB1), kinases (CDK2, PLK1) and a protease (MMP2). These data indicate a shift in response across time — with an increase in DNA repair, tissue remodeling and repression of cell proliferation acutely (24–72 h). Pathway-based integration (bottom up approach) identified common molecular and pathway responses to low dose radiation, including oxidative stress, nitric oxide signaling and transcriptional regulation through the SP1 factor that would not have been identified by the individual data sets. Significant regulation of key downstream metabolites of nitrative stress was measured within these pathways. Among the features identified in our study, the regulation of MMP2 and SP1 was experimentally validated. Our results demonstrate the advantage of data integration to broadly define the pathways and networks that represent the mechanisms by which complex biological systems respond to perturbation. - Highlights: • Low dose ionizing radiation altered homeostasis in 3D skin tissue model. • Global gene/protein/metabolite data integrated using complementary statistical approaches • Time and location-specific change in matrix regulation, ROS/RNS and DNA repair pathways detected • Low dose exposure alters metabolites involved in nitric oxide biosynthesis and wound healing. • Computationally predicted regulators of primary mechanisms were experimentally validated.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2025

OEDI—Solar Grid Integration Data and Analytics Library

As a part of the Open Energy Data Initiative, this effort aims to develop and demonstrate novel distribution state estimation, control optimization, and transient analysis as well as provide access to data, data integration, and mapping information. More specifically, the focus of the effort will be on physics-based distribution system state estimation, hybrid (physics-based and machine learning) distribution optimal power flow, and event detection/analysis for solar integration and analytics. This work will enable reproducible, robust, replicable, and generalizable R&D in simulation and emulation of solar system integration. These test models and datasets will provide an integrated library for developing and testing power system operation technologies. To make the library user-friendly, this project will provide data curation tools such as data translators, mapping scripts and APIs, database schemas and metadata, interfaces and user dashboard, source code for the reference algorithms, description of the use-cases/scenarios, and comprehensive information on all the assumptions.

14 SOLAR ENERGY↗
DOE OSTI2016

A Bloom Filter Based Scalable Data Integrity Check Tool for Large-Scale Dataset

Large scale HPC applications are becoming increasingly data intensive. At Oak Ridge Leadership Computing Facility (OLCF), we are observing the number of files curated under individual project are reaching as high as 200 millions and project data size is exceeding petabytes. These simulation datasets, once validated, often needs to be transferred to archival system for long term storage or shared with the rest of the research community. Ensuring the data integrity of the full dataset at this scale is paramount important but also a daunting task. This is especially true considering that most conventional tools are serial and file-based, unwieldy to use and/or can't scale to meet user's demand.To tackle this particular challenge, this paper presents the design, implementation and evaluation of a scalable parallel checksumming tool, fsum, which we developed at OLCF. It is built upon the principle of parallel tree walk and work-stealing pattern to maximize parallelism and is capable of generating a single, consistent signature for the entire dataset at extreme scale. We also applied a novel bloom-filter based technique in aggregating signatures to overcome the signature ordering requirement. Given the probabilistic nature of bloom filter, we provided a detailed error and trade-off analysis. Using multiple datasets from production environment, we demonstrated that our tool can efficiently handle both very large files as well as many small-file based datasets. Our preliminary test showed that on the same hardware, it outperforms conventional tool by as much as 4×. It also exhibited near-linear scaling properties when provisioned with more compute resources.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

BASIN-3D Data Integration for Selected ARM Data Field Campaign Report

The purpose of this data services request was to demonstrate integration of the Atmospheric Radiation Measurement (ARM) User Facility’s “met” datastreams with time series data from other earth science data sources using the BASIN-3D data synthesis software tool. BASIN-3D is an open-source Python library that enables researchers to integrate data across configured public and private data sources. It provides a common query language for researchers to request measurement locations and time series data based on specified locations, variables, time period, statistics, aggregation, and data quality. BASIN-3D acquires the data that match the query from each configured data source and translates the results into harmonized vocabularies, thus reducing researchers' data-wrangling effort. In addition, because the queries are executed on demand, researchers can easily regenerate their synthesized data sets as new data and/or data updates become available, eliminating one-off data products. BASIN-3D can output data using a variety of different data structures for end-user applications including Python pandas data frames and hdf5 output formats.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Towards Lightweight Data Integration Using Multi-Workflow Provenance and Data Observability

Modern large-scale scientific discovery requires multidisciplinary collaboration across diverse computing facilities, including High Performance Computing (HPC) machines and the Edge-to-Cloud continuum. Integrated data analysis plays a crucial role in scientific discovery, especially in the current AI era, by enabling Responsible AI development, FAIR, Reproducibility, and User Steering. However, the heterogeneous nature of science poses challenges such as dealing with multiple supporting tools, cross-facility environments, and efficient HPC execution. Building on data observability, adapter system design, and provenance, we propose MIDA: an approach for lightweight runtime Multi-workflow Integrated Data Analysis. MIDA defines data observability strategies and adaptability methods for various parallel systems and machine learning tools. With observability, it intercepts the dataflows in the background without requiring instrumentation while integrating domain, provenance, and telemetry data at runtime into a unified database ready for user steering queries. We conduct experiments showing end-to-end multi-workflow analysis integrating data from Dask and MLFlow in a real distributed deep learning use case for materials science that runs on multiple environments with up to 276 GPUs in parallel. We show near-zero overhead running up to 100,000 tasks on 1,680 CPU cores on the Summit supercomputer.

Santos Souza, Renan↗
DOE OSTI2023

Towards Integrating Data Quality Assessments and Radiometer Uncertainty for Determining the Expanded Uncertainty of Three-Component Solar Radiation Measurements

Accurate solar irradiance data are fundamental for determining the design and performance characteristics of photovoltaic systems. The uncertainty of solar irradiance measurements depends on many factors including radiometer design, calibration, installation, maintenance, and operational environment. The key contributors to this uncertainty can be classified as the measurement uncertainty of a particular radiometer and the operational uncertainty determined for the time of measurement. Radiometer measurement uncertainty estimates (U R ) can be based on well-established methods used as part of the radiometer calibration process. Estimates of operational uncertainties (U o ) require consideration of additional site-specific factors that affect data quality. A method is needed for establishing the accuracy of solar irradiance data by integrating an existing data quality process and measurement uncertainty estimates for specific radiometers. An algorithm has been developed to integrate data quality analyses and measurement uncertainty estimates for three-component solar irradiance data: global horizontal (total hemispheric) irradiance, direct normal (beam) irradiance, and diffuse horizontal (sky) irradiance collected at one- to 60-minute intervals. The algorithm has been tested using one-minute irradiance measurements. The goal of the project is to distribute a user-friendly software package based on the new algorithm.

data integrity↗
DOE OSTI2015

Integrated Data Collection Analysis (IDCA) Program - Statistical Analysis of RDX Standard Data Sets

The Integrated Data Collection Analysis (IDCA) program is conducting a Proficiency Test for Small- Scale Safety and Thermal (SSST) testing of homemade explosives (HMEs). Described here are statistical analyses of the results for impact, friction, electrostatic discharge, and differential scanning calorimetry analysis of the RDX Type II Class 5 standard. The material was tested as a well-characterized standard several times during the proficiency study to assess differences among participants and the range of results that may arise for well-behaved explosive materials. The analyses show that there are detectable differences among the results from IDCA participants. While these differences are statistically significant, most of them can be disregarded for comparison purposes to assess potential variability when laboratories attempt to measure identical samples using methods assumed to be nominally the same. The results presented in this report include the average sensitivity results for the IDCA participants and the ranges of values obtained. The ranges represent variation about the mean values of the tests of between 26% and 42%. The magnitude of this variation is attributed to differences in operator, method, and environment as well as the use of different instruments that are also of varying age. The results appear to be a good representation of the broader safety testing community based on the range of methods, instruments, and environments included in the IDCA Proficiency Test.

round-robin test↗
DOE OSTI2013

Integrated Data Collection Analysis (IDCA) Program — RDX Standard Data Sets

The Integrated Data Collection Analysis (IDCA) program is conducting a proficiency study for Small- Scale Safety and Thermal (SSST) testing of homemade explosives (HMEs). Described here are the results for impact, friction, electrostatic discharge, and differential scanning calorimetry analysis of the RDX Type II Class 5 standard, for a third and fourth time in the Proficiency Test and averaged with the analysis results from the first and second time. The results, from averaging all four sets (1, 2, 3 and 4) of data suggest a material to have slightly more impact sensitivity, more BAM friction sensitivity, less ABL friction sensitivity, similar ESD sensitivity, and same DSC sensitivity, compared to the results from Set 1, which was used previously as the values for the RDX standard in IDCA Analysis Reports.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

An integrated data management and informatics framework for continuous drug product manufacturing processes: A case study on two pilot plants

The pharmaceutical industry continuously looks for ways to improve its development and manufacturing efficiency. In recent years, such efforts have been driven by the transition from batch to continuous manufacturing and digitalization in process development. To facilitate this transition, integrated data management and informatics tools need to be developed and implemented within the framework of Industry 4.0 technology. Here, in this regard, the work aims to guide the data integration development of continuous pharmaceutical manufacturing processes under the Industry 4.0 framework, improving digital maturity and enabling the development of digital twins. This paper demonstrates two instances where a data integration framework has been successfully employed in academic continuous pharmaceutical manufacturing pilot plants. Details of the integration structure and information flows are comprehensively showcased. Approaches to mitigate concerns in incorporating complex data streams, including integrating multiple process analytical technology tools and legacy equipment, connecting cloud data and simulation models, and safeguarding cyber-physical security, are discussed. Critical challenges and opportunities for practical considerations are highlighted.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Towards physics-inspired data-driven weather forecasting: integrating data assimilation with a deep spatial-transformer-based U-NET in a case study with ERA5

Abstract. There is growing interest in data-driven weather prediction (DDWP), e.g., using convolutional neural networks such as U-NET that are trained on data from models or reanalysis. Here, we propose three components, inspired by physics, to integrate with commonly used DDWP models in order to improve their forecast accuracy. These components are (1) a deep spatial transformer added to the latent space of U-NET to capture rotation and scaling transformation in the latent space for spatiotemporal data, (2) a data-assimilation (DA) algorithm to ingest noisy observations and improve the initial conditions for next forecasts, and (3) a multi-time-step algorithm, which combines forecasts from DDWP models with different time steps through DA, improving the accuracy of forecasts at short intervals. To show the benefit and feasibility of each component, we use geopotential height at 500 hPa (Z500) from ERA5 reanalysis and examine the short-term forecast accuracy of specific setups of the DDWP framework. Results show that the spatial-transformer-based U-NET (U-STN) clearly outperforms the U-NET, e.g., improving the forecast skill by 45 %. Using a sigma-point ensemble Kalman (SPEnKF) algorithm for DA and U-STN as the forward model, we show that stable, accurate DA cycles are achieved even with high observation noise. This DDWP+DA framework substantially benefits from large (O(1000)) ensembles that are inexpensively generated with the data-driven forward model in each DA cycle. The multi-time-step DDWP+DA framework also shows promise; for example, it reduces the average error by factors of 2–3. These results show the benefits and feasibility of these three components, which are flexible and can be used in a variety of DDWP setups. Furthermore, while here we focus on weather forecasting, the three components can be readily adopted for other parts of the Earth system, such as ocean and land, for which there is a rapid growth of data and need for forecast and assimilation.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2014

ARM Data Integrator

The Atmospheric Radiation Measurement (ARM) Data Integrator (ADI) streamlines the development of scientific algorithms and analysis of time-series NetCDF data, and improves the content and consistency of the output data products produced by these algorithms. The framework automates the process of retrieving and preparing data for analysis, and allows users to design output data products through a graphical interface. It also provides a modular, flexible software development architecture that scientists can use to generate C, Python, and IDL source code templates that embed the pre and post processing logic allowing the scientist to focus on only their science. The input data, preprocessing, and output data specifications of algorithms are defined through a graphical interface and stored in a database. ADI implements workflow for data integration and supports user access to data through a library of software modules. Data preprocess capabilities supported include automated retrieval of data from input files, merging the retrieved data into appropriately sized chunks, and transformation of the data onto a common coordinate system grid. Through the graphical interface, users can view the details of both their data products and those in the ARM catalog and allows developers to use existing data product to build new data products. Views of the output data products include an overlay of how the design meets ARM archive's data standards providing the user with a visual cue indicating where their output violates an archive standard. The ADI libraries access the information provided through the GUI via a Postgres database. The ADI framework and its supporting components can significantly decrease the time and cost of implementing scientific algorithms while improving the ability of scientists to disseminate their results.

Source record↗
DOE OSTI2014

ARM Data Integrator

The Atmospheric Radiation Measurement (ARM) Data Integrator (ADI) streamlines the development of scientific algorithms and analysis of time-series NetCDF data, and improves the content and consistency of the output data products produced by these algorithms. The framework automates the process of retrieving and preparing data for analysis, and allows users to design output data products through a graphical interface. It also provides a modular, flexible software development architecture that scientists can use to generate C, Python, and IDL source code templates that embed the pre and post processing logic allowing the scientist to focus on only their science. The input data, preprocessing, and output data specifications of algorithms are defined through a graphical interface and stored in a database. ADI implements workflow for data integration and supports user access to data through a library of software modules. Data preprocess capabilities supported include automated retrieval of data from input files, merging the retrieved data into appropriately sized chunks, and transformation of the data onto a common coordinate system grid. Through the graphical interface, users can view the details of both their data products and those in the ARM catalog and allows developers to use existing data product to build new data products. Views of the output data products include an overlay of how the design meets ARM archive's data standards providing the user with a visual cue indicating where their output violates an archive standard. The ADI libraries access the information provided through the GUI via a Postgres database. The ADI framework and its supporting components can significantly decrease the time and cost of implementing scientific algorithms while improving the ability of scientists to disseminate their results.

Source record↗
DOE OSTI2019

Data Integration in Poplar: ‘Omics Layers and Integration Strategies

Populus trichocarpa is an important biofuel feedstock that has been the target of extensive research and is emerging as a model organism for plants, especially woody perennials. This research has generated several large ‘omics datasets. However, only few studies in Populus have attempted to integrate various data types. This review will summarize various ‘omics data layers, focusing on their application in Populus species. Subsequently, network and signal processing techniques for the integration and analysis of these data types will be discussed, with particular reference to examples in Populus.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2019

Method and application of data integration at a nuclear power plant. Light Water Reactor Sustainability Program report

The data of a nuclear power plant (NPP) is stored in silos for the various plant tools. A previous study conducted by the LWRS program identified the different data sources that can benefit online monitoring in an NPP. These data have different structures and tools and are therefore used independently. Integrating data from various sources can result in multiple benefits to online monitoring by enhancing the ability to compare data, reduce statistical uncertainty due to a larger data set, increase data heterogeneity in space and time to reveal confounded information, develop intelligent methods, and share data. Data integration can result in direct cost saving by reducing the amount of effort needed to find needed information. It can also enable establishing a data warehouse, the backbone repository of data for an NPP, and is necessary to evolve NPPs to a holistic online monitoring approach and automate labor-intensive activities. Whether within an NPP or for a fleet of NPPs, deploying a data warehouse in a plant or data broker for a fleet of plants requires overcoming multiple challenges, including the lack of a standard form of storing and using data and the lack of integration methods to compare and map the different data sources. This effort demonstrates two methods and applications for data integration through an application integration pilot and a data fusion pilot. The application integration is targeted towards the software interfacing of data and data management. This approach is commonly used by data integrators. The data fusion pilot is targeted towards intelligently mapping or coupling data for optimized data use. The pilots were performed in collaboration with two NPPs.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2013

Integrated Data Collection Analysis (IDCA) Program - RDX Standard Data Set 2

The Integrated Data Collection Analysis (IDCA) program is conducting a proficiency study for Small- Scale Safety and Thermal (SSST) testing of homemade explosives (HMEs). Described here are the results for impact, friction, electrostatic discharge, and differential scanning calorimetry analysis of the RDX Type II Class 5 standard, from testing the second time in the Proficiency Test. This RDX testing (Set 2) compared to the first (Set 1) was found to have about the same impact sensitivity, have more BAM friction sensitivity, less ABL friction sensitivity, similar ESD sensitivity, and same DSC sensitivity.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗