Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

BatAnalysis - A Comprehensive Python Pipeline for Swift BAT Survey Analysis

The Swift Burst Alert Telescope (BAT) is a coded-aperture gamma-ray instrument with a large field of view that primarily operates in survey mode when it is not triggering on transient events. The survey data consist of 80- channel detector plane histograms that accumulate photon counts over periods of at least 5 minutes. These histograms are processed on the ground and are used to produce the survey data set between 14 and 195 keV. Survey data comprise >90% of all BAT data by volume and allow for the tracking of long-term light curves and spectral properties of cataloged and uncataloged hard X-ray sources. Until now, the survey data set has not been used to its full potential due to the complexity associated with its analysis and the lack of easily usable pipelines. Here, we introduce the BatAnalysis Python package, a wrapper for HEASoftpy, which provides a modern, opensource pipeline to process and analyze BAT survey data. BatAnalysis allows members of the community to use BAT survey data in more advanced analyses of astrophysical sources, including pulsars, pulsar wind nebula, active galactic nuclei, and other known/unknown transient events that may be detected in the hard X-ray band. We outline the steps taken by the Python code and exemplify its usefulness and accuracy by analyzing survey data of the Crab Nebula, NGC 2992, and a previously uncataloged MAXI transient. The BatAnalysis package allows for ~18 yr of BAT survey data to be used in a systematic way to study a large variety of astrophysical sources.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2023

BatAnalysis - A Comprehensive Python Pipeline for Swift BAT Survey Analysis

The Swift Burst Alert Telescope (BAT) is a coded-aperture gamma-ray instrument with a large field of view that primarily operates in survey mode when it is not triggering on transient events. The survey data consist of 80-channel detector plane histograms that accumulate photon counts over periods of at least 5 minutes. These histograms are processed on the ground and are used to produce the survey data set between 14 and 195 keV. Survey data comprise >90% of all BAT data by volume and allow for the tracking of long-term light curves and spectral properties of cataloged and uncataloged hard X-ray sources. Until now, the survey data set has not been used to its full potential due to the complexity associated with its analysis and the lack of easily usable pipelines. Here, we introduce the BatAnalysis Python package, a wrapper for HEASoftpy, which provides a modern, open-source pipeline to process and analyze BAT survey data. BatAnalysis allows members of the community to use BAT survey data in more advanced analyses of astrophysical sources, including pulsars, pulsar wind nebula, active galactic nuclei, and other known/unknown transient events that may be detected in the hard X-ray band. We outline the steps taken by the Python code and exemplify its usefulness and accuracy by analyzing survey data of the Crab Nebula, NGC 2992, and a previously uncataloged MAXI transient. The BatAnalysis package allows for ~18 yr of BAT survey data to be used in a systematic way to study a large variety of astrophysical sources.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

System and methods for hardware-software cooperative pipeline error detection

An error reporting system utilizes a parity checker to receive data results from execution of an original instruction and a parity bit for the data. A decoder receives an error correcting code (ECC) for data resulting from execution of a shadow instruction of the original instruction, and data error correction is initiated on the original instruction result on condition of a mismatch between the parity bit and the original instruction result, and the decoder asserting a correctable error in the original instruction result.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Modeling Framework for the Assessment of a Sustainable Hydrogen Production and Supply Chain Network in California

The cost-effective and sustainable deployment of hydrogen supply and demand networks, especially in large economic regions like California, can be challenging considering the spatial-temporal availability and variability of the different actors across the network such as production processes, distribution modes, and end-users. In this presentation, we will provide an overview and demonstration of a modeling framework used to assess the environmental, economic, and human health impacts of plausible hydrogen production and supply chain networks in California. Scenarios focus on green hydrogen production pathways using water electrolysis and biomass gasification. End-use applications included in the model are transit, medium and heavy-duty trucking, port authorities, and power and aviation companies that currently consume natural gas, diesel, and aviation fuel for their day-to-day operation. Representative locations for hydrogen production and end-use are based on recent projections of the hydrogen economy in California. All mass and energy flows, as well as estimated emissions, are based on H2A process model designs and projections of technology performance, literature review, and LBNL process, economic and life cycle modeling, and not on company data for the sake of this presentation. Human health impacts are included following methodologies developed for the University of California Irvine HyDeal project. Life cycle phases associated with hydrogen production include feedstock preparation (water and biomass), energy production and consumption (renewable, grid, and combination of renewable and grid electricity), maintenance (chemical utilization in electrolysis and natural gas combustion in gasification), carbon sequestration, hydrogen storage (compression and liquefaction), and distribution (truck and pipeline). We apply the framework utilizing California specific emission factors, financial data, and human health damages and explore the impact of network characteristics on results. Example variations include: the inclusion of policy incentives or not, different representations of the electricity grid and source, electrolysis versus gasification versus combinations of both for production, liquefaction versus compression based on producer capacity cutoffs, transportation truck versus pipeline based on existing infrastructure, and ultimate end use. Comparison of these different scenarios can help inform future projects by demonstrating the trade-offs among environmental, economic, and human health impacts. This model, automated in R, is a starting platform upon which new analysis, modeling capabilities, locations, and emission factors can be rapidly tested and integrated.

Zaki, Mohammed Tamim↗
DOE OSTI2025

DESI 2024 III: baryon acoustic oscillations from galaxies and quasars

We present the DESI 2024 galaxy and quasar baryon acoustic oscillations (BAO) measurements using over 5.7 million unique galaxy and quasar redshifts in the range 0.1 < z < 2.1. Divided by tracer type, we utilize 300,017 galaxies from the magnitude-limited Bright Galaxy Survey with 0.1 < z < 0.4, 2,138,600 Luminous Red Galaxies with 0.4 < z < 1.1, 2,432,022 Emission Line Galaxies with 0.8 < z < 1.6, and 856,652 quasars with 0.8 < z < 2.1, over a ∼ 7,500 square degree footprint. The analysis was blinded at the catalog-level to avoid confirmation bias. All fiducial choices of the BAO fitting and reconstruction methodology, as well as the size of the systematic errors, were determined on the basis of the tests with mock catalogs and the blinded data catalogs. We present several improvements to the BAO analysis pipeline, including enhancing the BAO fitting and reconstruction methods in a more physically-motivated direction, and also present results using combinations of tracers. We employ a unified BAO analysis method across all tracers. We present a re-analysis of SDSS BOSS and eBOSS results applying the improved DESI methodology and find scatter consistent with the level of the quoted SDSS theoretical systematic uncertainties. With the total effective survey volume of ∼ 18 Gpc3, the combined precision of the BAO measurements across the six different redshift bins is ∼0.52%, marking a 1.2-fold improvement over the previous state-of-the-art results using only first-year data. We detect the BAO in all of these six redshift bins. The highest significance of BAO detection is 9.1σ at the effective redshift of 0.93, with a constraint of 0.86% placed on the BAO scale. We find that our observed BAO scales are systematically larger than the prediction of the Planck 2018-ΛCDM at z < 0.8. We translate the results into transverse comoving distance and radial Hubble distance measurements, which are used to constrain cosmological models in our companion paper.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Addressing delayed case reporting in infectious disease forecast modeling

Infectious disease forecasting is of great interest to the public health community and policymakers, since forecasts can provide insight into disease dynamics in the near future and inform interventions. Due to delays in case reporting, however, forecasting models may often underestimate the current and future disease burden. In this paper, we propose a general framework for addressing reporting delay in disease forecasting efforts with the goal of improving forecasts. We propose strategies for leveraging either historical data on case reporting or external internet-based data to estimate the amount of reporting error. We then describe several approaches for adapting general forecasting pipelines to account for under- or over-reporting of cases. We apply these methods to address reporting delay in data on dengue fever cases in Puerto Rico from 1990 to 2009 and to reports of influenza-like illness (ILI) in the United States between 2010 and 2019. Through a simulation study, we compare method performance and evaluate robustness to assumption violations. Our results show that forecasting accuracy and prediction coverage almost always increase when correction methods are implemented to address reporting delay. Some of these methods required knowledge about the reporting error or high quality external data, which may not always be available. Provided alternatives include excluding recently-reported data and performing sensitivity analysis. This work provides intuition and guidance for handling delay in disease case reporting and may serve as a useful resource to inform practical infectious disease forecasting efforts.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

“Innervated” Pipelines: A New Technology Platform for In-Situ Repair and Embedded Intelligence

The overall vision pursued under the project would ultimately enable capability for real-time operational monitoring of natural gas (and other) pipeline infrastructures through the combination of in-situ repair and rehabilitation methods with embedded fiber optic sensing and associated data analytics methods and platform tools. Benefits to the public include more resilient and robust natural gas pipeline infrastructures with the potential to expand the applications of developed technologies under the program to other important areas of critical energy infrastructure in the future such as H2 pipelines and infrastructure, municipal and other civil infrastructure, as well as subsurface oil, gas, and geothermal infrastructure applications. The program has successfully demonstrated feasibility for all of the critical enabling elements and serves as a foundation for future technology maturation, deployment, and customization for other potential applications and high-priority needs into the future. Several underlying innovations have shown potential for commercial deployment and are the subject of continued development and technology transfer based upon patents submitted during the project duration.

02 PETROLEUM↗
DOE OSTI2025

UnigeneFinder: An Automated Pipeline for Gene Calling From Transcriptome Assemblies Without a Reference Genome

ABSTRACT For most species, transcriptome data are much more readily available than genome data. Without a reference genome, gene calling is cumbersome and inaccurate because of the high degree of redundancy in de novo transcriptome assemblies. To simplify and increase the accuracy of de novo transcriptome assembly in the absence of a reference genome, we developed UnigeneFinder. Combining several clustering methods, UnigeneFinder substantially reduces the redundancy typical of raw transcriptome assemblies. This pipeline offers an effective solution to the problem of inflated transcript numbers, achieving a closer representation of the actual underlying genome. UnigeneFinder performs comparably or better, compared with existing tools, on plant species with varying genome complexities. UnigeneFinder is the only available transcriptome redundancy solution that fully automates the generation of primary transcript, coding region, and protein sequences, analogous to those available for high‐quality reference genomes. These features, coupled with the pipeline’s cross‐platform implementation, focus on automation, and an accessible, user‐friendly interface, make UnigeneFinder a useful tool for many downstream sequence‐based analyses in nonmodel organisms lacking a reference genome, including differential gene expression analysis, accurate ortholog identification, functional enrichments, and evolutionary analyses. UnigeneFinder also runs efficiently both on high‐performance computing (HPC) systems and personal computers, further reducing barriers to use.

Xue, Bo [Plant Resilience Institute Michigan State↗
DOE OSTI2020

Multi-Functional Distributed Fiber Sensors for Pipeline Monitoring and Methane Detections. Final Report

As an abundant and cheap fossil energy source, natural gas has become a significant energy supply to support the United States’ economy. However, the large-scale extraction and utilization of natural gas also impose significant challenges on methane leakage. This problem is exacerbated by aging gas utility delivery systems, including interstate high-pressure pipelines, storage, and transmission facilities. This project aims to develop a cost-effective fiber optical sensing method that can perform multi-parameter real-time measurements of natural gas pipelines across long interrogation distances up to 100 km with 1-meter spatial resolution. This sensing tool can evaluate overall pipeline efficiency and reduce methane emissions for mid-stream methane infrastructures. To accomplish this objective, research and development efforts funded by this project have resulted in the following accomplishments: This project successfully has developed new functional sensory polymer materials using Metal-Organic Frameworks (MOFs) that can be coated on optical fiber through the reel-to-reel coating process. Functional polymer-coated optical fibers can perform sensitive methane detection through evanescence wave interaction and strain-based measurements to achieve 1% detection sensitivities. The new sensors fibers support both distributed measurements and multiplexed fiber sensors array for multi-point measurements. This project developed and optimized a new multi-core optical fiber that supports simultaneous and distributed measurements of strain and temperatures with 1-meter spatial resolutions across up to 100-km interrogation distance. This new fiber, combined with sensory polymercoated fiber, could perform both distributed temperature and methane detections. This project developed a new artificial intelligence big data algorithm approach that can effectively analyze high-resolution data harnessed by distributed fiber sensors to protect natural gas pipelines against external threats and detect internal defects induced by corrosion. Working with our industry partner, this project developed new optical fibers that support fiber sensor fabrications through polymer coating after the fibers are drawn. These new fibers eliminate the need for direct sensor fabrication when the fiber is fabricated on a fiber draw tower, which drastically expands fiber sensors' applicability. This research project has significantly advanced the distributed fiber sensing technology. It will dramatically increase the applicability and adaptability of distributed fiber sensors for a wide array of applications in energy, sustainability, and environmental science, including structural health monitoring of natural gas pipelines, oil infrastructures, hydrogen facilities, and environmental monitoring of carbon storage sites, water supply systems, and others.

03 NATURAL GAS↗
DOE OSTI2025

Site B - NREL ASSIST (SN11) Thermodynamic Retrievals TROPoe / Derived Data

This dataset contains daily files with thermodynamic profiles retrieved with the optimal estimation physical retrieval TROPoe v0.12 (Turner and Löhnert 2014; Turner and Blumberg 2019; Turner and Löhnert 2021). The profiles are retrieved every 10 minutes from instantaneous observations from the NREL ASSIST-II (SN 11) infrared spectrometer. Observations are noise-filtered but not averaged in time to minimize errors due to non-uniform clouds. Additional input data in TROPoe are cloud base height (CBH), which is a combined data product that uses data from ceilometers at sites A1 and H and scanning lidars from ARM sites C1 and E37. The CBH is weighted inversely proportionally to the distance to the respective site to take into account the spatial variability of clouds (see https://github.com/StefanoWind/ASSIST_analysis/blob/main/awaken_processing/combine_cbh.py). The full pipeline for running the retrieval is available at https://github.com/StefanoWind/TROPoe_processor. Met data was not ingested. In addition to these temporally resolved input data, TROPoe requires an a priori dataset (prior) that provides mean climatological estimates of thermodynamic profiles and specifies how temperature and humidity covary with height as an input (for details see, e.g., Djalalova et al. 2022). The prior is a key component of the retrieval and provides a constraint on the ill-posed inversion problem. A monthly prior was computed from operational radiosonde launches at ARM SGP, OK.

17 WIND ENERGY↗
DOE OSTI2025

Site G - NREL ASSIST (SN10) Thermodynamic Retrievals TROPoe / Derived Data

This dataset contains daily files with thermodynamic profiles retrieved with the optimal estimation physical retrieval TROPoe v0.12 (Turner and Löhnert 2014; Turner and Blumberg 2019; Turner and Löhnert 2021). The profiles are retrieved every 10 minutes from instantaneous observations from the NREL ASSIST-II (SN 10) infrared spectrometer. Observations are noise-filtered but not averaged in time to minimize errors due to non-uniform clouds. Additional input data in TROPoe are cloud base height (CBH), which is a combined data product that uses data from ceilometers at sites A1 and H and scanning lidars from ARM sites C1 and E37. The CBH is weighted inversely proportionally to the distance to the respective site to take into account the spatial variability of clouds (see https://github.com/StefanoWind/ASSIST_analysis/blob/main/awaken_processing/combine_cbh.py). The full pipeline for running the retrieval is available at https://github.com/StefanoWind/TROPoe_processor. Met data was not ingested. In addition to these temporally resolved input data, TROPoe requires an a priori dataset (prior) that provides mean climatological estimates of thermodynamic profiles and specifies how temperature and humidity covary with height as an input (for details see, e.g., Djalalova et al. 2022). The prior is a key component of the retrieval and provides a constraint on the ill-posed inversion problem. A monthly prior was computed from operational radiosonde launches at ARM SGP, OK.

17 WIND ENERGY↗
DOE OSTI2025

Site C1a - NREL ASSIST (SN12) Thermodynamic Retrievals TROPoe / Derived Data

This dataset contains daily files with thermodynamic profiles retrieved with the optimal estimation physical retrieval TROPoe v0.12 (Turner and Löhnert 2014; Turner and Blumberg 2019; Turner and Löhnert 2021). The profiles are retrieved every 10 minutes from instantaneous observations from the NREL ASSIST-II (SN 12) infrared spectrometer. Observations are noise-filtered but not averaged in time to minimize errors due to non-uniform clouds. Additional input data in TROPoe are cloud base height (CBH), which is a combined data product that uses data from ceilometers at sites A1 and H and scanning lidars from ARM sites C1 and E37. The CBH is weighted inversely proportionally to the distance to the respective site to take into account the spatial variability of clouds (see https://github.com/StefanoWind/ASSIST_analysis/blob/main/awaken_processing/combine_cbh.py). The full pipeline for running the retrieval is available at https://github.com/StefanoWind/TROPoe_processor. Met data was not ingested. In addition to these temporally resolved input data, TROPoe requires an a priori dataset (prior) that provides mean climatological estimates of thermodynamic profiles and specifies how temperature and humidity covary with height as an input (for details see, e.g., Djalalova et al. 2022). The prior is a key component of the retrieval and provides a constraint on the ill-posed inversion problem. A monthly prior was computed from operational radiosonde launches at ARM SGP, OK.

17 WIND ENERGY↗
DOE OSTI2025

Prediction of Silicon Content in a Blast Furnace via Machine Learning: A Comprehensive Processing and Modeling Pipeline

Silicon content plays an important role in determining the operational efficiency of blast furnaces (BFs) and their downstream processes in integrated steelmaking; however, existing sampling methods and first-principles models are somewhat limited in their capability and flexibility. Current data-based prediction models primarily rely on a limited set of manually selected furnace parameters. Additionally, different BFs present a diverse set of operating parameters and state variables that are known to directly influence the hot metal’s silicon content, such as fuel injection, blast temperature, and raw material charge composition, among other process variables that have their own impacts. The expansiveness of the parameter set adds complexity to parameter selection and processing. This highlights the need for a comprehensive methodology to integrate and select from all relevant parameters for accurate silicon content prediction. Providing accurate silicon content predictions would enable operators to adjust furnace conditions dynamically, improving safety and reducing economic risk. To address these issues, a two-stage approach is proposed. First, a generalized data processing scheme is proposed to accommodate diverse furnace parameters. Second, a robust modeling pipeline is used to establish a machine learning (ML) model capable of predicting hot metal silicon content with reasonable accuracy. The method employed herein predicted the average Si content of the upcoming furnace cast with an accuracy of 91% among 200 target predictions for a specific furnace provisioned by the XGBoost model. This prediction is achieved using only the past shift’s operating conditions, which should be available in real time. This performance provides a strong baseline for the modeling approach with potential for further improvement through provision of real-time features.

Chemistry↗
DOE OSTI2026

HydraGNN_GFM_FineTuning4Materials v1.0

This repository enables fine-tuning of the HydraGNN Predictive GFM 2026 — an open-source ensemble of pre-trained graph foundation models for atomistic materials modeling, developed at Oak Ridge National Laboratory. The GFM 2026 is freely available and downloadable via Globus from the OLCF Data Constellation (DOI: 10.13139/OLCF/2562660). Starting from these pre-trained weights, this repository provides a complete transfer learning pipeline for adapting the GFM ensemble to domain-specific molecular and materials property prediction tasks. It includes: 1) Utilities for ensemble fine-tuning with task-specific output heads 2) Example pipelines for eight widely-used materials and molecular datasets 3) Tools for model adaptation and head configuration 4) Data preprocessing utilities for each supported dataset 5) Benchmarking and evaluation scripts

Ungerboeck, Linda↗
DOE OSTI2022

Development of integrated screening, cultivar optimization, and verification research (DISCOVR): A coordinated research-driven approach to improve microalgal productivity, composition, and culture stability for commercially viable biofuels production

To address major knowledge gaps and barriers to the commercial development of algal biomass for biofuels and co-products, a collaborative consortium, Development of Integrated Screening, Cultivar Optimization, and Verification Research (DISCOVR), was established in 2016. Funded by the U.S. Department of Energy (DOE) Bioenergy Technologies Office (BETO), this consortium constitutes a partnership between four DOE national laboratories - Pacific Northwest National Laboratory (PNNL), Los Alamos National Laboratory (LANL), the National Renewable Energy Laboratory (NREL), and Sandia National Laboratories (SNL) - and the Arizona Center for Algae Technology and Innovation (AzCATI) at Arizona State University. To address the barriers of strain selection for achieving high seasonal productivities with a suitable composition and culture resilience, a tiered strain down-selection pipeline is implemented. At Tier I, the temperature and salinity tolerance of strains is determined in flask cultures; at Tier II, the areal biomass productivity and composition is determined in climate-simulation photobioreactors; at Tier III, the productivity and culture stability are determined in outdoor raceways. The top performing strains move forward to long-term testing at the algae testbed site at AzCATI to generate annual biomass productivity data. Concurrent to the strain down-selection in the DISCOVR pipeline, hypotheses for increasing biomass productivity, shifting biomass composition to enhance intrinsic value, and improving culture stability and resistance to pests are also tested. Techno-economic analyses are carried out to determine whether promising findings from laboratory studies or proposed modifications in outdoor pond cultivation conditions translate into reductions in the minimum biomass selling price (MBSP). Notably, in the three years following the launch of DISCOVR, annual biomass productivity has increased from 11.7 to 17.6 g m -2 day -1 , resulting in an MBSP decrease from 824 to 611 $ ton -1 .

09 BIOMASS FUELS↗
DOE OSTI2024

Solar PV Oil and Gas Pipeline Setbacks: Ordinances (2022) and Extrapolated Trends

This dataset represents solar energy setback requirements from oil and gas pipelines. A setback requirement is a minimum distance from a pipeline that an energy project may be developed. As of April 2022, no ordinances were discovered for any counties. Such ordinances are likely to arise as regulations continue to expand. Therefore, this dataset applies a 30-meter setback, sourced from trends in other infrastructure. A TIF data file and a PNG map of the data are provided, showing areas where solar energy is prohibited or permitted across the contiguous United States. For further details and citation, please refer to the publication linked below: Lopez, Anthony, Pavlo Pinchuk, Michael Gleason, Wesley Cole, Trieu Mai, Travis Williams, Owen Roberts, Marie Rivers, Mike Bannister, Sophie-Min Thomson, Gabe Zuckerman, and Brian Sergi. 2024. Solar Photovoltaics and Land-Based Wind Technical Potential and Supply Curves for the Contiguous United States: 2023 Edition. Golden, CO: National Renewable Energy Laboratory. NREL/TP-6A20-87843.

Array↗
DOE OSTI2022

Gap Analysis on the Impacts of Hydrogen Addition to the North American Natural Gas Infrastructure Polyethylene Pipelines

The existing natural gas pipeline system is being considered as a means to distribute blends of clean hydrogen and natural gas as one component of clean hydrogen’s role in enabling a future low-carbon energy system. With nearly three million miles of transmission and distribution lines in the United States, this national asset is uniquely situated as a transition strategy to assist in the decarbonization of the US economy. 1 This report reviews existing data on the effects of hydrogen on polyethylene (PE) natural gas pipeline materials and identifies gaps where additional investigation is required to fully assess deleterious impacts to the polyethylene pipes from exposure to hydrogen. This gap analysis will assist in determining the feasibility of blending hydrogen gas into existing natural gas pipelines.

08 HYDROGEN↗