Search NASA⌕ Search

SEARCH · Search NASA

Results for “data pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Intelligent experiments through real-time AI: Fast Data Processing and Autonomous Detector Control for sPHENIX and future EIC detectors

This R&D project, initiated by the DOE Nuclear Physics AI-Machine Learning initiative in 2022, leverages AI to address data processing challenges in high-energy nuclear experiments (RHIC, LHC, and future EIC). Our focus is on developing a demonstrator for real-time processing of high-rate data streams from sPHENIX experiment tracking detectors. The limitations of a 15 kHz maximum trigger rate imposed by the calorimeters can be negated by intelligent use of streaming technology in the tracking system. The approach efficiently identifies low momentum rare heavy flavor events in high-rate p+p collisions (3MHz), using Graph Neural Network (GNN) and High Level Synthesis for Machine Learning (hls4ml). Success at sPHENIX promises immediate benefits, minimizing resources and accelerating the heavy-flavor measurements. The approach is transferable to other fields. For the EIC, we develop a DIS-electron tagger using Artificial Intelligence - Machine Learning (AI-ML) algorithms for real-time identification, showcasing the transformative potential of AI and FPGA technologies in high-energy nuclear and particle experiments real-time data processing pipelines.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

A Gigaparsec-scale Hydrodynamic Volume Reconstructed with Deep Learning

The next generation of spectroscopic surveys will map the large-scale structure of the Universe at high redshifts (2 ≤ z ≤ 5) using millions of quasar spectra, enabling major advances in constraining both the standard cosmological model and its extensions. Robust cosmological analyses of these data sets require numerical simulations that both cover gigaparsec volumes and resolve features on ∼10 kpc scales and smaller. However, running such large-volume, high-resolution hydrodynamic simulations is computationally prohibitive. We present a generative deep learning model that enhances a low-resolution, gigaparsec-scale (960 h −1 Mpc) hydrodynamic simulation using a smaller (80 h −1 Mpc) high-resolution input hydrodynamic simulation as training data. The resulting enhanced simulation reproduces the line-of-sight power spectrum to within ∼10% and the three-dimensional power spectrum at the ∼20% level at intermediate to small scales (k ≲ 2 h Mpc −1 ). Our method shows strong promise for producing realistic simulations for cosmological analyses with current surveys such as the Dark Energy Spectroscopic Instrument and upcoming next-generation experiments, but further improvements are needed to accurately recover the large-scale modes. We publicly release the enhanced hydrodynamic simulation, along with a halo catalog from a companion N-body dark matter simulation to support the calibration of data analysis pipelines for these large-scale surveys.

Convolutional neural networks↗
DOE OSTI2025

SITCOMTN-149: An Interim Report on the LSSTComCam On-Sky Campaign

From 24 October to 11 December 2024, the NSF-DOE Vera C. Rubin Observatory conducted an on-sky campaign using the engineering LSST Commissioning Camera (LSSTComCam) to test the end-to-end functionality of hardware and software, as well as operational procedures. This interim report provides a preliminary technical overview of our understanding of the integrated system performance based tests and analyses conducted during the LSSTComCam on-sky campaign. The objectives are to synthesize what we have learned about the system in a timely way to inform on-going commissioning efforts, and to inform the Rubin science community on the progress of the LSSTComCam on-sky campaign. The report is organized into sections to describe major activities during the campaign, as well as multiple aspects of the demonstrated system and science performance. All of the results presented here are to be understood as work in progress using engineering data and the initial versions of the data processing pipelines; the report is a living document that will be updated as analyses are refined.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

Evaluating Direct and Indirect Influence on EV Charging Stations Across the US

The adoption of new technology for electric vehicles (EV) and mobility applications can bring underappreciated vulnerabilities to the power grid. One area of potential fraud and adversarial influence is through the business ecosystem of startups that own and deploy EV technology. Yet, there are no models or analyses that map the network of organizations and people that have direct and indirect influence over technologies currently deployed in the grid. To fill this gap, we develop a multilayer network model to measure direct and indirect influence on EV charging stations. First, we create and adversarial socio-technical network (ASTN) model via a data fusion pipeline for different US regions of interest (ROI). Then, we develop an integrated ASTN for Chicago, Los Angeles, New York, and Philadelphia. We rank EV charging companies direct influence within each geographic region as well as indirect influence via social network analysis. While some companies have strong direct and indirect influence (i.e., ChargePoint) others show a mismatch between their influence over charging stations and their position within the social network. For example, Tesla has strong direct influence on stations and weak indirect influence over competitors. In contrast, 7Charge has weak direct influence over stations, but strong indirect influence over competitors.

29 - ENERGY PLANNING, POLICY AND ECONOMY↗
DOE OSTI2026

Cold Weather Impacts on Electric School Bus Performance in Aurora, Colorado

This brief highlight details the key takeaways from a project that utilized NLR's Fleet Research, Energy Data, and Insights (FleetREDI) data analysis pipeline related to electric school bus (ESB) operation. ESBs using battery energy as their primary heating source have a higher energy consumption rate in cold weather, which fleet managers can account for when planning ESB purchases and making dispatching and charging decisions. Researchers found that electric school buses operate 2-5 times more efficiently than conventional buses, on average. Cold weather can double electric school bus energy demands, but strategies such as thermal pre-conditioning significantly reduce this effect. Understanding these impacts can help fleets plan charging, dispatching, and purchase decisions.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2026

Exploration of Real Time Inference for MI-RR Deblending on GPU/TPU Systems

The Fermilab Main Injector (MI) and Recycler Ring (RR) share a common beam loss monitor (BLM) system, making loss events difficult to attribute to their source machine when beam is present in both simultaneously. The Real-time Edge AI for Distributed Systems (READS) project addresses this by deblending BLM readings in real time using machine learning (ML). The current FPGA based implementation meets the sub-3 ms latency requirement but carries a resource intensive hls4ml development cycle, motivating exploration of GPU based deployment. This paper characterizes inference latency on an NVIDIA Jetson Orin Nano and introduces a packet organization scheme for assembling synchronized event frames from seven distributed BLM DAQ streams. Using a Python based DAQ simulation with injected timing jitter in place of unavailable live beam data, the pipeline achieved an average end to end latency of 0.456 ms (σ = 0.122 ms) across 167,000 test frames, comfortably meeting the timing constraint. Early outliers were attributed to TensorRT warm-up rather than steady state limitations, suggesting GPU based inference is a viable alternative to the existing FPGA implementation.

Yu, Kellen [Fermilab; Cornell U.]↗
DOE OSTI2026

Virtual Inspection of Advanced Manufacturing via Process-Scale Digital Twins (Abbreviated Report)

Inspection and certification comprise the most significant bottlenecks in advanced manufacturing for NNSA applications, often requiring far more time and resources than the fabrication of the parts themselves. Traditional methods, such as manual review and X-ray computed tomography, are not only slow and costly, but also struggle to provide a clear connection between manufacturing instructions and the final performance of critical components. This gap limits both the agility and assurance needed to support the modernization and safety of the United States nuclear stockpile. In response, our Strategic Initiative established a digital twin framework that integrates realtime process monitoring, automated data analysis, and immersive virtual reality collaboration into a unified inspection pipeline. By leveraging data from sensors, machine instructions, and imaging, we created high-fidelity virtual models of manufactured parts that could be rapidly analyzed and certified. This approach was first demonstrated with Direct Ink Write, and then extended to other manufacturing settings, including conventional (or “subtractive”) manufacturing and to predict the end of life performance of parts per the aging and lifetimes programs. The result is a transformational capability: inspection times have been reduced by a factor of 120,000 without loss of accuracy and while simultaneously improving traceability and confidence in part quality. This framework not only streamlines certification for critical applications, but also positions the national security enterprise to respond more flexibly to emerging challenges, supporting agile manufacturing and digital engineering practices across a broad range of mission-relevant domains.

42 ENGINEERING↗
DOE OSTI2025

A Performant, Scalable Processing Pipeline for High‐Quality and FAIR Environmental Sensor Data

High-resolution environmental monitoring is necessary to record, understand, and predict biogeochemical and ecological changes particularly in coastal systems but brings significant challenges in processing and making rapidly available the resulting data. The COMPASS-FME project established a network of coastal observational sites across the Chesapeake Bay and western Lake Erie regions extensively instrumented with soil, vegetation, and weather sensors logging data every 15 min. Our data processing framework, written in R and completely open source, prioritizes rapid model-experiment iteration and makes biogeochemical data rapidly available for quality assurance/quality control, analysis, and model ingestion. This pipeline is distinguished by a standardized and modular approach to data curation, extensive metadata and documentation, and its high performance. These attributes combine to make biogeochemical data rapidly accessible across COMPASS-FME and the broader community. Flexible, powerful, and reproducible approaches to handling high-volume environmental data are crucial for accelerating biogeosciences research.

Pennington, Stephanie C. [Pacific Northwest Nation↗
DOE OSTI2025

DoCeph: DPU-Offloaded Messaging in Ceph for Reduced Host CPU Utilization

Ceph is a widely used distributed object store, but its messenger layer imposes substantial CPU overhead on the host. To address this limitation, we propose DoCeph, a DPU-offloaded storage architecture for Ceph that disaggregates the system by offloading the communication-intensive messaging component to the DPU while retaining the storage backend on the host. The DPU efficiently manages communication, using lightweight RPC for metadata operations and DMA for data transfer. Moreover, DoCeph introduces a pipelining technique that overlaps data transmission with buffer preparation, mitigating hardware-imposed transfer size limitations. We implemented DoCeph on a Ceph cluster with NVIDIA BlueField-3 DPUs. Evaluation results indicate that DoCeph cuts host CPU usage by up to 92% while sustaining stable throughput and providing larger performance benefits for object writes over 1 MB.

Park, Kuri [Sogang University]↗
DOE OSTI2025

The Dynamic Networks Experiments: Virtual Experiments to Quantify Gains in Nuclear Explosion Monitoring

We describe an ongoing series of virtual experiments conducted collaboratively by four United States National Laboratories: Sandia National Laboratories, Los Alamos National Laboratory, Lawrence Livermore National Laboratory, and Pacific Northwest National Laboratory. These Dynamic Network Experiments (DNEs) provide an experimental framework to evaluate the potential impact of new research tools on nuclear explosion monitoring. The second DNE (DNE2), completed in 2024, exploited waveform data (seismic, infrasound, and electromagnetic) that was recorded by multi-modal sensors within and near the Nevada National Security Site and synthetic radionuclide signatures over multiple time periods. During the execution of DNE2, we processed and analyzed data through a multi-stage event processing pipeline that ingested raw data, performed quality control, detected signals, built events from these signals, located these events, and characterized the events’ source types and sizes. For each stage and over the entire event processing pipeline, we evaluated performance changes by comparing the performance of new data processing methods, models, and algorithms against a baseline. We also performed an additional execution phase to assess event processing pipeline function, speed, and efficiency against that of an expert analyst, including computational and manual efforts. Finally, we assessed the impact and effort of modern computing infrastructure on the monitoring pipeline. This paper describes key elements of the DNEs, from formulation through execution, as demonstrated in DNE2. The DNEs introduce several novel concepts to quantitatively measure the potential impact of new methods on explosion monitoring, including the collaborative design of multi-modal datasets, performance and logistical metrics, and integrated analyses.

42 ENGINEERING↗
DOE OSTI2024

Benefits and Limits of Phasing Alleles for Network Inference of Allopolyploid Complexes

Abstract Accurately reconstructing the reticulate histories of polyploids remains a central challenge for understanding plant evolution. Although phylogenetic networks can provide insights into relationships among polyploid lineages, inferring networks may be hindered by the complexities of homology determination in polyploid taxa. We use simulations to show that phasing alleles from allopolyploid individuals can improve phylogenetic network inference under the multispecies coalescent by obtaining the true network with fewer loci compared with haplotype consensus sequences or sequences with heterozygous bases represented as ambiguity codes. Phased allelic data can also improve divergence time estimates for networks, which is helpful for evaluating allopolyploid speciation hypotheses and proposing mechanisms of speciation. To achieve these outcomes in empirical data, we present a novel pipeline that leverages a recently developed phasing algorithm to reliably phase alleles from polyploids. This pipeline is especially appropriate for target enrichment data, where the depth of coverage is typically high enough to phase entire loci. We provide an empirical example in the North American Dryopteris fern complex that demonstrates insights from phased data as well as the challenges of network inference. We establish that our pipeline (PATÉ: Phased Alleles from Target Enrichment data) is capable of recovering a high proportion of phased loci from both diploids and polyploids. These data may improve network estimates compared with using haplotype consensus assemblies by accurately inferring the direction of gene flow, but statistical nonidentifiability of phylogenetic networks poses a barrier to inferring the evolutionary history of reticulate complexes.

Evolutionary Biology↗
DOE OSTI2024

Optimizing Distributed Training on Frontier for Large Language Models

Large language models (LLMs) have demonstrated remarkable success as foundational models, benefiting various downstream applications through fine-tuning. Loss scaling studies have demonstrated the superior performance of larger LLMs compared to their smaller counterparts. Nevertheless, training LLMs with billions of parameters poses significant challenges and requires considerable computational resources. For example, training a one trillion parameter GPT-style model on 20 trillion tokens requires a staggering 120 million exaflops. This research explores efficient distributed training strategies to extract this computation from Frontier, the world's first exascale supercomputer. We enable and investigate various model and data parallel training techniques, such as tensor parallelism, pipeline parallelism, and sharded data parallelism, to facilitate training a trillion-parameter model on Frontier. We empirically assess these techniques and their associated parameters to determine their impact on memory footprint, communication latency, and GPU's computational efficiency. We analyze the complex interplay among these techniques and find a strategy to combine them to achieve high throughput through hyperparameter tuning. We have identified efficient strategies for training large LLMs of varying sizes through empirical analysis and hyperparameter tuning. For 22 Billion, 175 Billion, and 1 Trillion parameters, we achieved GPU throughputs of 38.38%, 36.14%, and 31.96%, respectively. For the training of the 175 Billion parameter model and the 1 Trillion parameter model, we achieved 100% weak scaling efficiency on 1024 and 3072 Mi250X GPUs, respectively. We also achieved strong scaling efficiencies of 89% and 87% for these two models. We trained these models only tens of iterations instead of training till completion.

Yin, Junqi↗
DOE OSTI2026

Adaptive Cybersecurity for Distributed Energy Resources (AdCyDER): Online Reinforcement Learning with Stackelberg-Optimized Defenses — Pipeline Architecture, Evaluation Methodology, and Findings from a Synthetic-Data Evaluation

This report documents the design and evaluation of an integrated online-learning pipeline developed within the AdCyDER project for Distributed Energy Resource (DER) cybersecurity. The pipeline couples a Reinforcement Learning (RL) attack classifier — which produces an attack-type probability distribution — with a Stackelberg game-theoretic (GT) defense selector that consumes those distributions alongside SME-encoded priors over (defense, attack) effectiveness pairings and perdefense costs to choose grid-health-preserving defenses. The objective is not attack classification per se but production of distributions that drive effective defense selection through the Stackelberg layer, learned from delayed grid-health feedback rather than labeled attack data. AdCyDER as a whole is broader than the work presented here; this report covers the specific RL/GT loop integration and its evaluation. We present the integrated pipeline (SCADA telemetry with Fronius inverter physics, Suricata IDS, time-windowed aggregation, per-facility LSTM classifier, Stackelberg optimizer, OpenC2 actuators), an experimental campaign of 28 eight-hour iterations across three baseline modes, and a pipeline-ordered diagnostic protocol. The protocol identifies two distinct failure modes within the loop: paired supervised ceilings on the same features establish that the deployed online RL classifier (macro F1 ≈ 0.07) sits at least 4.7× below a same-architecture supervised LSTM (≈ 0.34) and 10–11× below a linear feature-signal ceiling (≈ 0.70–0.79 depending on per-facility isolation), localizing the dominant failure to the training procedure; and the reward signal driving online updates carries weak directional coupling with classifier correctness in the methodology-expected direction (multi-lens convergent: top-decile P(true) records produce more frequent state changes and slightly larger improvements, top-vs-bot Cohen’s 𝑑 ≈ −0.19), but at effect magnitudes too small to drive gradient-based learning at the campaign sample size. The original learning hypothesis is not supported by the data. The primary contributions are the diagnostic methodology — proposed as a transferable falsification protocol for online RL/GT defense pipelines learning from delayed environmental reward — and the open, reproducible experimental infrastructure. We outline reward reformulation as the highest-priority aspirational next step given the underpowered-but-aligned Q6 reading, with hardware-in-the-loop evaluation as the broadest scope-expansion option.

Blakely, Benjamin [Argonne National Laboratory (AN↗
DOE OSTI2024

Testing the LSST Difference Image Analysis Pipeline Using Synthetic Source Injection Analysis

Abstract We evaluate the performance of the Legacy Survey of Space and Time Science Pipelines Difference Image Analysis (DIA) on simulated images. By adding synthetic sources to galaxies on images, we trace the recovery of injected synthetic sources to evaluate the pipeline on images from the Dark Energy Science Collaboration Data Challenge 2. The pipeline performs well, with efficiency and flux accuracy consistent with the signal-to-noise ratio of the input images. We explore different spatial degrees of freedom for the Alard–Lupton polynomial-Gaussian image subtraction kernel and analyze for trade-offs in efficiency versus artifact rate. Increasing the kernel spatial degrees of freedom reduces the artifact rate without loss of efficiency. The flux measurements with different kernel spatial degrees of freedom are consistent. We also here provide a set of DIA flags that substantially filter out artifacts from the DIA source table. We explore the morphology and possible origins of the observed remaining subtraction artifacts and suggest that given the complexity of these artifact origins, a convolution kernel with a set of flexible bases with spatial variation may be needed to yield further improvements.

Liu, S. (ORCID:0000000244612143)↗
DOE OSTI2025

Determination of Constraint-Independent Crack Tip Opening Angle for Stable Crack Growth in High-Strength Ductile Steels

The crack tip opening angle (CTOA) is one of fracture toughness parameters that has been used for decades in describing large stable crack growth in thin-walled aerospace structures under the low-constraint conditions. Recently, the pipeline industry has developed a growing interest in using the CTOA parameter to serve as the minimum required fracture toughness to arrest dynamic crack propagation in modern gas transmission pipelines made of high-strength ductile steel. To meet this industrial need, the CTOA test standard ASTM E3039 was therefore developed for measuring the constant critical CTOA. ASTM E3039 recommends a drop weight tearing test (DWTT) specimen with a shallow crack for standard CTOA testing, but its CTOA may depend on the low constraint condition of the DWTT specimen at the crack tip. Verifying the constraint independence of the DWTT-measured CTOA thus becomes indispensable for applying CTOA toughness to the running fracture control in the pipeline design. For this purpose, the present paper evaluates critical CTOA values in a set of fracture toughness tests on single-edge notched bend (SENB) specimens with shallow and deep cracks, based on four CTOA estimation models. Among these, the Ln(P)-LLD linear fit model is similar to that recommended by ASTM E3039 for CTOA calculation. Fracture test data for X80 pipeline steel and HY80 structural steel were considered in the CTOA evaluation. The results showed that the four CTOA models were able to determine a constraint independent CTOA value for stable crack growth in the SENB specimens. As a result, a single, reliable, constant CTOA value could be determined regardless of the specimen geometry or the crack-tip constraint conditions. Therefore, the CTOA measured using ASTM E3039 is constraint-independent and transferable to use in cases of actual cracks propagating in gas transmission pipelines.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Towards constraining cosmological parameters with SPT-3G observations of 25% of the sky

The South Pole Telescope (SPT), using its third-generation camera, SPT-3G, is conducting observations of the cosmic microwave background (CMB) in temperature and polarization across approximately 10 000 deg$^2$ of the sky at 95, 150, and 220 GHz. This comprehensive dataset should yield stringent constraints on cosmological parameters. In this work, we explore its potential to address the Hubble tension by forecasting constraints from temperature, polarization, and CMB lensing on early dark energy (EDE) and the variation in electron mass in spatially flat and curved universes. For this purpose, we investigate first whether analyzing the distinct SPT-3G observation fields independently, as opposed to as a single, unified region, results in a loss of information relevant to cosmological parameter estimation. We develop a realistic temperature and polarization likelihood pipeline capable of analyzing these fields in these two ways, and subsequently forecast constraints on cosmological parameters. Our findings indicate that any loss of constraining power from analyzing the fields separately is primarily concentrated at low multipoles ($\ell$ < 50) and the overall impact on the relative uncertainty on standard $Λ$ cold dark matter parameters is minimal (< 3%). Our forecasts suggest that SPT-3G data should improve by more than a factor of 90 and 190 the figure of merit of the EDE and the varying electron mass models, respectively, when combined with Planck data. The likelihood pipeline developed and used in this work is made publicly available online.

Vitrier, A. [Paris, Inst. Astrophys.]↗
DOE OSTI2024

Data Format and Descriptions for the Alabama Carbon Storage: Data Sharing and Engagement Project

The Alabama Carbon Storage: Data Sharing and Engagement (ACS-DSE) project seeks to develop publicly accessible geologic carbon storage models and data across the southern Gulf Coastal Plain of Alabama. The public online platform developed for this project will include geologic, geophysical, infrastructure, and other relevant datasets and geologic models of the study area. Datasets, model surfaces (e.g. structural contour maps, isolith maps, porosity maps), and infrastructure data (e.g. offshore pipelines, field boundaries) will be downloadable in commonly used file formats. The anticipated primary geologic datasets are well headers, formation tops, average reservoir properties, and core analyses; these will be available as commaseparated values (CSV) text files and MS Excel workbooks. Geophysical logs will be available in Log ASCII Standard (LAS) file format. Modeled surfaces, such as structure contour maps, will be available in ArcGIS formats and text files. Infrastructure data will be available as ArcGIS shapefiles. This document provides information on the data sources and attributes of the datasets.

01 COAL, LIGNITE, AND PEAT↗