Search NASA⌕ Search

SEARCH · Search NASA

Results for “data pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Cognitive analysis of metabolomics data for systems biology

Cognitive computing is revolutionizing the way big data are processed and integrated, with artificial intelligence (AI) natural language processing (NLP) platforms helping researchers to efficiently search and digest the vast scientific literature. Most available platforms have been developed for biomedical researchers, but new NLP tools are emerging for biologists in other fields and an important example is metabolomics. NLP provides literature-based contextualization of metabolic features that decreases the time and expert-level subject knowledge required during the prioritization, identification and interpretation steps in the metabolomics data analysis pipeline. Here, we describe and demonstrate four workflows that combine metabolomics data with NLP-based literature searches of scientific databases to aid in the analysis of metabolomics data and their biological interpretation. Additionally, the four procedures can be used in isolation or consecutively, depending on the research questions. The first, used for initial metabolite annotation and prioritization, creates a list of metabolites that would be interesting for follow-up. The second workflow finds literature evidence of the activity of metabolites and metabolic pathways in governing the biological condition on a systems biology level. The third is used to identify candidate biomarkers, and the fourth looks for metabolic conditions or drug-repurposing targets that the two diseases have in common. The protocol can take 1–4 h or more to complete, depending on the processing time of the various software used.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

Planck 2018 Results - II. Low Frequency Instrument data processing

We present a final description of the data-processing pipeline for the Planck Low Frequency Instrument (LFI), implemented for the 2018 data release. Several improvements have been made with respect to the previous release, especially in the calibration process and in the correction of instrumental features such as the effects of nonlinearity in the response of the analogue-to-digital converters. We provide a brief pedagogical introduction to the complete pipeline, as well as a detailed description of the important changes implemented. Self-consistency of the pipeline is demonstrated using dedicated simulations and null tests. We present the final version of the LFI full sky maps at 30, 44, and 70 GHz, both in temperature and polarization, together with a refined estimate of the solar dipole and a final assessment of the main LFI instrumental parameters.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

Reinforcement Learning for In-Spill Optimization of the Mu2e Resonant Extraction: Compensating Non-Stationarity

We present design considerations and challenges for the fast machine learning component of a third-order resonant beam extraction regulation system being commissioned to deliver steady beam rates to the mu2e experiment at Fermilab. Dedicated quadrupoles drive the tune toward the 29/3 resonance each spill, extracting beam at kV multiwire septa. The overall Spill Regulation System consists of (1) a “slow” process using ~100-spill averages to adjust the base quad ramp infrequently, (2) a feedforward harmonic content compensator, and (3) the “fast” ML agent reacting during each ongoing spill with on-the-fly additive corrections to the sum of (1) and (2). We have demonstrated improved beam-rate steadying for a fast ML agent compared to a PID controller using a quasi-physical spill simulation, and demonstrated distillation of that simulation into a predictive surrogate model. Current work includes a data-and-training pipeline to generate data-aware surrogates with real-world dynamics, even as the dynamics shift unpredictably. The surrogates are to act as RL environments against which to train our fast ML control agents before deploying them on FPGA in the live system. Further current efforts focus on modeling and controlling beam loss around the storage ring, understanding additional available hardware inputs to the model, and the interplay of these with beam-steadying performance.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2023

Contextually aware roadside radiation measurement testbed

Here we demonstrate a contextually aware multimodal roadside radiation measurement detection testbed for traffic monitoring applications in nuclear nonproliferation. Many variables in traffic such as vehicle or cargo size, mass, speed, shape, and distance of closest approach can have significant impacts on the radiation measured from a vehicle-transported radiation source. These factors can lead to uncertainties in the analysis of the radiation source, especially for lower-strength radiation sources of interest. Our testbed, known as the Multimodal Measurement System (MMS) uses non-radiation sensors including magnetometers, geophones, radiofrequency receivers, cameras, and LiDAR to extract contextual information about vehicles passing by the system. These contextual data can then be fused with data from radiation measurements to increase the system’s sensitivity and accuracy in nuclear threat detection applications. This work describes the instrumentation of the MMS and its data acquisition pipeline. Furthermore, we describe the pre-analysis performed on the raw multimodal data streams for data fusion, and the high-level machine learning analyses for detection and characterization. The variety of sensors within the MMS provides a valuable testbed that can be used to identify the combinations of contextual sensors that provide the greatest improvements to radiation source detection and characterization within the restrictions for various proliferation detection applications. The MMS is also modular so that additional combinations of sensors can be explored in the future.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2020

FLEET: Flexible Efficient Ensemble Training for Heterogeneous Deep Neural Networks

Parallel training of an ensemble of Deep Neural Networks (DNN) on a cluster of nodes is an effective approach to shorten the process of neural network architecture search and hyper-parameter tuning for a given learning task. Prior efforts have shown that data sharing, where the common preprocessing operation is shared across the DNN training pipelines, saves computational resources and improves pipeline efficiency. Data sharing strategy, however, performs poorly for a heterogeneous set of DNNs where each DNN has varying computational needs and thus different training rate and convergence speed. This paper proposes FLEET, a flexible ensemble DNN training framework for efficiently training a heterogeneous set of DNNs. We build FLEET via several technical innovations. We theoretically prove that an optimal resource allocation is NP-hard and propose a greedy algorithm to efficiently allocate resources for training each DNN with data sharing. We integrate data-parallel DNN training into ensemble training to mitigate the differences in training rates and introduce checkpointing into this context to address the issue of different convergence speeds. Experiments show that FLEET significantly improves the training efficiency of DNN ensembles without compromising the quality of the result.

Guan, Hui↗
DOE OSTI2024

TomoPyUI : a user-friendly tool for rapid tomography alignment and reconstruction

The management and processing of synchrotron and neutron computed tomography data can be a complex, labor-intensive and unstructured process. Users devote substantial time to both manually processing their data ( i.e. organizing data/metadata, applying image filters etc. ) and waiting for the computation of iterative alignment and reconstruction algorithms to finish. In this work, we present a solution to these problems: TomoPyUI , a user interface for the well known tomography data processing package TomoPy . This highly visual Python software package guides the user through the tomography processing pipeline from data import, preprocessing, alignment and finally to 3D volume reconstruction. The TomoPyUI systematic intermediate data and metadata storage system improves organization, and the inspection and manipulation tools (built within the application) help to avoid interrupted workflows. Notably, TomoPyUI operates entirely within a Jupyter environment. Herein, we provide a summary of these key features of TomoPyUI , along with an overview of the tomography processing pipeline, a discussion of the landscape of existing tomography processing software and the purpose of TomoPyUI , and a demonstration of its capabilities for real tomography data collected at SSRL beamline 6-2c.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2026

Carbon dioxide pipeline network transportation cost model: evaluating economic and geographic factors for efficient carbon capture, storage, and utilization

This study presents a comprehensive pipeline network modeling framework to estimate the CO 2 delivery cost for CO 2 utilization and geologic CO 2 storage across the United States. We developed a Python-based CO 2 pipeline transportation cost model leveraging Argonne National Laboratory’s pipeline engineering expertise and detailed natural gas transmission pipeline cost data across U.S. regions. Using existing road corridors as practical routing guides, the model designs pipeline networks that aggregate CO 2 from one or multiple sources and deliver it to selected destinations. It then minimizes the total transportation cost by optimizing pipeline diameters and incorporating booster pumps. A key contribution is the incorporation of up-to-date, region-specific cost factors with itemized components for materials, labor, miscellaneous construction expenses, and right-of-way acquisition. Results emphasize that regional variation and economies of scale associated with CO 2 pipeline costs are significant and should be explicitly accounted for in screening and planning studies. By combining realistic routing constraints with regionalized cost inputs, the model provides transparent design methodology and location-specific insights into source–destination delivery costs, including the effects of routing complexity along existing road networks. We demonstrate the model with two illustrative case studies – one for CO 2 storage and one for CO 2 utilization – in which the model designs pipeline networks spanning hundreds of miles across the states, collecting CO 2 from multiple sources and delivering it to designated endpoints while minimizing levelized cost of delivery via diameter and compression optimization. The model offers a practical, scalable approach for alternative design option screening and early-stage CO 2 transportation planning.

CCS↗
DOE OSTI2022

A Simple Standard for Sharing Ontological Mappings (SSSOM)

Abstract Despite progress in the development of standards for describing and exchanging scientific information, the lack of easy-to-use standards for mapping between different representations of the same or similar objects in different databases poses a major impediment to data integration and interoperability. Mappings often lack the metadata needed to be correctly interpreted and applied. For example, are two terms equivalent or merely related? Are they narrow or broad matches? Or are they associated in some other way? Such relationships between the mapped terms are often not documented, which leads to incorrect assumptions and makes them hard to use in scenarios that require a high degree of precision (such as diagnostics or risk prediction). Furthermore, the lack of descriptions of how mappings were done makes it hard to combine and reconcile mappings, particularly curated and automated ones. We have developed the Simple Standard for Sharing Ontological Mappings (SSSOM) which addresses these problems by: (i) Introducing a machine-readable and extensible vocabulary to describe metadata that makes imprecision, inaccuracy and incompleteness in mappings explicit. (ii) Defining an easy-to-use simple table-based format that can be integrated into existing data science pipelines without the need to parse or query ontologies, and that integrates seamlessly with Linked Data principles. (iii) Implementing open and community-driven collaborative workflows that are designed to evolve the standard continuously to address changing requirements and mapping practices. (iv) Providing reference tools and software libraries for working with the standard. In this paper, we present the SSSOM standard, describe several use cases in detail and survey some of the existing work on standardizing the exchange of mappings, with the goal of making mappings Findable, Accessible, Interoperable and Reusable (FAIR). The SSSOM specification can be found at http://w3id.org/sssom/spec. Database URL: http://w3id.org/sssom/spec

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2021

dbprocessing

dbprocessing is a python framework for automating data processing pipelines. The dbprocessing package uses a configuration file to define the input, intermediate, and output data products for a particular data type as well as the processes that connect those data products. In addition, dbprocessing uses python scripts called inspectors to determine if a particular file matches a configured data product. If a particular data product is found, it is ingested into the dbprocessing database (sqlite3 or postgres) which then triggers all of the chained processes to the final data product. While the framework is written in python, it can run software in any language but may require a “wrapper” to translate the dbprocessing command line arguments to the form expected by the software.

Walker, Andrew↗
DOE OSTI2025

The DECADE cosmic shear project I: A new weak lensing shape catalog of 107 million galaxies

We present the Dark Energy Camera All Data Everywhere (DECADE) weak lensing dataset: a catalog of 107 million galaxies observed by the Dark Energy Camera (DECam) in the northern Galactic cap. This catalog was assembled from public DECam data including survey and standard observing programs. These data were consistently processed with the Dark Energy Survey Data Management pipeline as part of the DECADE campaign and serve as the basis of the DECam Local Volume Exploration survey (DELVE) Early Data Release 3 (EDR3). We apply the Metacalibration measurement algorithm to generate and calibrate galaxy shapes. After cuts, the resulting cosmology-ready galaxy shape catalog covers a region of $5,\!412 \,\,{\rm deg}^2$ with an effective number density of $4.59\,\, {\rm arcmin}^{-2}$. The coadd images used to derive this data have a median limiting magnitude of $r = 23.6$, $i = 23.2$, and $z = 22.6$, estimated at ${\rm S/N} = 10$ in a 2 arcsecond aperture. We present a suite of detailed studies to characterize the catalog, measure any residual systematic biases, and verify that the catalog is suitable for cosmology analyses. In parallel, we build an image simulation pipeline to characterize the remaining multiplicative shear bias in this catalog, which we measure to be $m = (-2.454 \pm 0.124) \times10^{-2}$ for the full sample. Despite the significantly inhomogeneous nature of the data set, due to it being an amalgamation of various observing programs, we find the resulting catalog has sufficient quality to yield competitive cosmological constraints.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2023

NSRDB (National Solar Radiation Database Software) [SWR-23-77]

This repository contains all of the methods for the NSRDB data processing pipeline. The NSRDB is a serially complete collection of hourly and half-hourly values of meteorological data and the three most common measurements of solar radiation: global horizontal, direct normal and diffuse horizontal irradiance.

Benton, Brandon↗
DOE OSTI2026

Scalable GPS Data Logging To Support Advanced Fleet Analysis

This highlight details the key takeaways from a project that utilized NLR's Fleet Research, Energy Data, and Insights (FleetREDI) data analysis pipeline. National Laboratory of the Rockies researchers developed and demonstrated low-cost, open-source Arduino data loggers with 3D-printed cases that are compatible with global navigational systems and built with components available ubiquitously worldwide, enabling cost-effective collection and analysis of fleet operational data. Validated on an overseas transit bus fleet, NLR analysis showed that, with sufficient charging opportunities, 90% of observed duty cycles could be accomplished by electric buses with no modifications to operations.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2023

Spatio-Temporal Denoising Graph Autoencoders with Data Augmentation for Missing Photovoltaic Data Imputation

The integration of the global Photovoltaic (PV) market with real time data-loggers has enabled large scale PV data analytical pipelines for power forecasting and long-term reliability assessment of PV fleets. Nevertheless, the performance of PV data analysis heavily depends on the quality of PV timeseries data. This paper proposes a novel Spatio-Temporal Denoising Graph Autoencoder (STD-GAE) framework to impute missing PV Power Data. STDGAE exploits temporal correlation, spatial coherence, and value dependencies from domain knowledge to recover missing data. It is empowered by two modules. (1) To cope with sparse yet various scenarios of missing data, STD-GAE incorporates a domain-knowledge aware data augmentation module that creates plausible variations of missing data patterns. This generalizes STD-GAE to robust imputation over different seasons and environment. (2) STD-GAE nontrivially integrates spatiotemporal graph convolution layers (to recover local missing data by observed “neighboring” PV plants) and denoising autoencoder (to recover corrupted data from augmented counterpart) to improve the accuracy of imputation accuracy at PV fleet level. We have evaluated our proposed model on two realworld PV datasets. Experimental results show that STD-GAE can achieve a gain of 43.14% in imputation accuracy and remains less sensitive to missing rate, different seasons, and missing scenarios, compared with state-of-the-art data imputation methods such as MIDA and LRTC-TNN.

Fan, Yangxin↗
DOE OSTI2024

Estimating Electrification Potential for Class 8 Regional-Haul Trucks

This one-page highlight details the key takeaways from a project that utilized NREL's Fleet Research, Energy Data, and Insights (FleetREDI) data analysis pipeline. As part of the North American Council for Freight Efficiency's (NACFE's) Run on Less Depot data workshop, NREL sought to understand how Tesla semi-trucks would perform in real-world regional haul applications. Analysis reveals that the modeled Tesla trucks, with an average efficiency of 1.78 kWh/mi, struggle to achieve full operational coverage using current battery and charging configurations assuming operations remain unchanged. However, in an extreme case where ubiquitous charging exists, 100% EV coverage is possible for the given drive cycles. These findings highlight the trade-off between battery size and charge rate in electrification potential and emphasize the necessity for advancements in charging infrastructure to enable electric trucks for regional haul operations.

ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2023

Dataset Repository for Investigating Suicide Risk Using Social and Environmental Determinants of Health

Suicide is frequently modeled as a function of genetics and environment, where the latter refers to factors other than direct biological consequences, such as air quality, financial level, social connectivity, transportation and food access, and homelessness status. According to the World Health Organization, clean air, a stable climate, adequate water, sanitation and hygiene, safe chemical use, radiation protection, healthy and safe workplaces, sound agricultural practices, health-supportive cities and built environments, and a preserved natural environment are all prerequisites for good health. Understanding the relationships between these determinants and mental health outcomes requires standardized data that can be included in healthcare programs and health outcome models. There is a wealth of publicly available data on social and environmental factors provided by various US organizations that can benefit the design of health care systems and public health interventions, as well as improve our comprehension of factors that impact health. Such information would not only help improve the understanding of individual and community risk but also identify new risk factors that have not previously been therapeutically targeted, especially in terms of their impact on mental health. However, curating and standardizing such datasets is challenging because they are often recorded at numerous geographical and temporal resolutions and with varying spatial and temporal granularities. To address this challenge, we launched an endeavor in conjunction with the Veterans Health Administration to collect publicly available socioeconomic and environmental determinants of health statistics in the US. In this manuscript, we describe a social and environmental determinants of health (SEDH) datasets repository, data curation documentation, and a pipeline framework for data generation; This effort started in 2020, when we began constructing a scalable pipeline to automate the download, extraction, preparation, analysis, and production of datasets. These datasets have been made available to the VHA and may be shared upon agreement with collaborating organizations.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2023

MaNGA galaxy properties – II. A detailed comparison of observed and simulated spiral galaxy scaling relations

ABSTRACT We present a catalogue of dynamical properties for 2368 late-type galaxies from the MaNGA survey. The latter complements the catalogue of photometric properties for the same sample based on deep optical dark energy sky instrument legacy imaging survey (DESI) photometry processed with autoprof . Rotation curves (RCs), extracted by model-fitting H α velocity maps from the MaNGA Data Analysis Pipeline, extend out to 1.4 (1.9) Re for the primary (secondary) MaNGA samples, respectively. The RCs and ancillary MaNGA Pipe3D data products were used to construct various fundamental galaxy scaling relations that are also compared uniformly with similar relations from numerical investigation of a hundred astrophysical object (NIHAO) zoom-in simulations. Simulated NIHAO galaxies were found to broadly reproduce the observed MaNGA galaxy population for $\log (M_*/{\rm {\rm M}_{\odot })\, \gt\, 8.5}$. Some discrepancies remain, such as those pertaining to central stellar densities and the diversity of RCs due to strong feedback schemes. Also presented are spatially resolved scatters for the velocity-size-stellar mass (VRM*) structural relations using MaNGA and NIHAO samples. The scatter for these relations in the galaxian interiors is a consequence of the diversity of inner RC shapes, while scatter in the outskirts is dictated by the large range of stellar surface densities, which is itself driven by sporadic star formation. The detailed, spatially resolved scatter analysis highlights the complex interplay between local and global astrophysical processes and provides a strong constraint to numerical simulations.

Astronomy & Astrophysics↗
DOE OSTI2024

The FRB-searching Pipeline of the Tianlai Cylinder Pathfinder Array

This paper presents the design, calibration, and survey strategy of the Fast Radio Burst (FRB) digital backend and its real-time data processing pipeline employed in the Tianlai Cylinder Pathfinder Array. The array, consisting of three parallel cylindrical reflectors and equipped with 96 dual-polarization feeds, is a radio interferometer array designed for conducting drift scans of the northern celestial semi-sphere. The FRB digital backend enables the formation of 96 digital beams, effectively covering an area of approximately 40 square degrees with the 3 dB beam. Our pipeline demonstrates the capability to conduct an automatic search of FRBs, detecting at quasi-real-time and classifying FRB candidates automatically. The current FRB searching pipeline has an overall recall rate of 88%. During the commissioning phase, we successfully detected signals emitted by four well-known pulsars: PSR B0329+54, B2021+51, B0823+26, and B2020+28. We report the first discovery of an FRB by our array, designated as FRB 20220414A. We also investigate the optimal arrangement for the digitally formed beams to achieve maximum detection rate by numerical simulation.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2026

RTN-117: Image Calibration and Instrument Signal Removal for the First Year of the LSST

The NSF-DOE Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) requires calibration products that provide uniform, stable, and accurate photometric and astrometric performance across the 3.2 gigapixel focal plane and throughout the 10-year survey. This paper details the algorithms and workflows used to produce instrument calibrations and remove instrumental artifacts for Data Preview 2 (DP2)---the first end-to-end processing demonstration using on-sky data with the LSST Camera (LSSTCam). We describe the verification, acceptance, and certification framework used to assess calibration quality and quantify residual systematics. We show baseline metrics on calibrated science images to evaluate the robustness of the calibration and instrument signature removal (ISR) data processing pipelines for DP2. Finally, we summarize the known limitations observed in DP2 production and outline expected algorithmic improvements for the first public LSST data release (Data Release~1, DR1).

79 ASTRONOMY AND ASTROPHYSICS↗