Search NASA⌕ Search

SEARCH · Search NASA

Results for “pipeline infrastructure”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Updated Natural Gas Pathways in R&D GREET 2024 Rev.1

Natural gas (NG) is a relatively low-cost fossil fuel with vast infrastructure in the United States (U.S.) making it easier to transport and store compared to other fossil fuels such as coal and petroleum. In 2023, the U.S. consumed 32.5 trillion cubic feet or 33.6 Quad Btu of NG, accounting for 36% of the nation’s total primary energy consumption. NG is commonly used for electricity generation, industrial applications, commercial and residential activities and transportation purposes, as shown in Figure 1. This widespread reliance on NG underscores the need to assess its full environmental impact, requiring careful analysis of the entire supply chain from production (i.e., recovery, gathering and boosting [G&B], and processing of raw gas) to final delivery to end users (e.g., via pipelines).

03 NATURAL GAS↗
DOE OSTI2026

Opportunities in AI/ML for the Rubin LSST Dark Energy Science Collaboration

The Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST) will produce unprecedented volumes of heterogeneous astronomical data (images, catalogs, and alerts) that challenge traditional analysis pipelines. The LSST Dark Energy Science Collaboration (DESC) aims to derive robust constraints on dark energy and dark matter from these data, requiring methods that are statistically powerful, scalable, and operationally reliable. Artificial intelligence and machine learning (AI/ML) are already embedded across DESC science workflows, from photometric redshifts and transient classification to weak lensing inference and cosmological simulations. Yet their utility for precision cosmology hinges on trustworthy uncertainty quantification, robustness to covariate shift and model misspecification, and reproducible integration within scientific pipelines. This white paper surveys the current landscape of AI/ML across DESC's primary cosmological probes and cross-cutting analyses, revealing that the same core methodologies and fundamental challenges recur across disparate science cases. Since progress on these cross-cutting challenges would benefit multiple probes simultaneously, we identify key methodological research priorities, including Bayesian inference at scale, physics-informed methods, validation frameworks, and active learning for discovery. With an eye on emerging techniques, we also explore the potential of the latest foundation model methodologies and LLM-driven agentic AI systems to reshape DESC workflows, provided their deployment is coupled with rigorous evaluation and governance. Finally, we discuss critical software, computing, data infrastructure, and human capital requirements for the successful deployment of these new methodologies, and consider associated risks and opportunities for broader coordination with external actors.

Aubourg, Eric [APC, Paris] (ORCID:000000025592023X↗
DOE OSTI2022

Moving small files in a networked environment

Globally distributed computing infrastructures, such as clouds and supercomputers, are currently used to manage data that is generated with an unprecedented speed from a variety of resources. Coping with this trend, the volume of data exchanged across distant sites increases substantially. To accelerate data transfer, high-speed networks are provided to connect remote sites. Most existing data movement solutions are optimized for moving large files. However, it is still challenging to transfer a large number of small files across networks. This disadvantage not only lowers data transfer performance, but also decreases overall system utilization. Here, we identify that moving small files is mainly constrained by degraded file system throughput, not just network performance as might be suspected. We have built a data transfer pipeline model to analyze the impact of small network I/O and storage I/O on data movement. Extending one of the widely used open source data movement solutions, GridFTP, we demonstrate several appropriate engineering approaches that mitigate the bottleneck and increase data transfer efficiency. We show optimizations that improve data transfer performance more than 5 times. In comparison to existing solutions, our approaches can save a significant amount of system resources for moving lots of small files.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Grid Operator Analytics and Assessment Tools for Inverter- Based Resources Dominated Grid (GOAAT-IBR) Project Update

This presentation provides an update on the OPTIMA GOAAT project, with emphasis on the cloud-native data platform developed in-house to ingest, manage, and operationalize high-resolution power system data. Since our last NASPI presentation, accessible via OSTI ID #2671437, the project team advanced the design and deployment of a scalable architecture capable of handling both synchronized and non-synchronized streams, including PMU, point-on-wave (POW), COMTRADE, and SCADA data. These materials review the project status, recent progress, and key lessons learned. The core of the presentation examines the architecture and engineering of our cloud-native ingestion and data management platform. We then explain how pipelines were designed to collect, normalize, time-align, store, and serve heterogeneous data at scale. We will discuss design choices such as data models, streaming versus batch ingestion, storage tiers, and interoperability with analytics applications. Practical experiences with cloud-native technologies were shared during the event, including benefits, limitations, and integration challenges in a utility environment, along with methods used to improve performance, reduce latency, and optimize resource usage. The presentation also showcases user interface designs and visualization tools that convert raw measurements and analytics results into intuitive, actionable insights for operators and engineers. During the presentation examples were provided demonstrating how visualization, event views, and summarized analytics enhance situational awareness and support operational decision-making. These use cases illustrate how a well-designed data infrastructure can bridge the gap between high-volume measurements and practical grid operations.

Aminifar, Farrokh↗
DOE OSTI2020

FALCON: Framework for Anomaly Detection in Industrial Control Systems

Industrial Control Systems (ICS) are used to control physical processes in critical infrastructure. These systems are used in a wide variety of operations such as water treatment, power generation and distribution, and manufacturing. While the safety and security of these systems are of serious concern, recent reports have shown an increase in targeted attacks aimed at manipulating physical processes to cause catastrophic consequences. This trend emphasizes the need for algorithms and tools that provide resilient and smart attack detection mechanisms to protect ICS. In this paper, we propose an anomaly detection framework for ICS based on a deep neural network. The proposed methodology uses dilated convolution and long short-term memory (LSTM) layers to learn temporal as well as long term dependencies within sensor and actuator data in an ICS. The sensor/actuator data are passed through a unique feature engineering pipeline where wavelet transformation is applied to the sensor signals to extract features that are fed into the model. Additionally, this paper explores four variations of supervised deep learning models, as well as an unsupervised support vector machine (SVM) model for this problem. The proposed framework is validated on Secure Water Treatment testbed results. This framework detects more attacks in a shorter period of time than previously published methods.

97 - MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Cooling and Timing Tests of the ATLAS Fast TracKer VME Boards

The Fast TracKer (FTK) is an ATLAS trigger upgrade built for full-event, low-latency, high-rate tracking. The FTK core, made of 9U VME boards, performs the most demanding computational task. The associative memory board (AMB) serial link processor and the auxiliary card (AUX), plugged on the front and back sides of the same VME slot, constitute the processing unit (PU), which finds tracks using hits from eight layers of the inner detector. The PU works in pipeline with the second stage board (SSB), which finds 12-layer tracks by adding extra hits to the identified tracks. In the designed configuration, 16 PUs and four SSBs are installed in a VME crate. The high power consumption of the AMB, AUX, and SSB (respectively, of about 250, 70, and 160 W per board) required the development of a custom cooling system. Even though the expected power consumption for each VME crate of the FTK system is high compared with a common VME setup, the 8 FTK core crates will use ≈60 kW, which is just a fraction of the power and the space needed for a CPU farm performing the same task. We report on the integration of 32 PUs and eight SSBs inside the FTK system, on the infrastructures needed to run and cool them, and on the tests performed to verify the system processing rate and the temperature stability at a safe value.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2025

Efficient Synthetic Natural Gas Production from Direct Air Capture Using Titania-Based Dual Function Materials

Converting atmospheric CO2 into methane offers a compelling pathway to store intermittent renewable electricity and enhance energy security using existing natural gas infrastructure. However, current CO2 utilization approaches remain energy- and capital-intensive, largely due to the need for separate capture, purification, and conversion steps. Integrating CO2 capture with catalytic methanation represents a transformative strategy for process intensification on both the unit operation and molecular levels. We report a series of Ru-Na/TiO2 dual function materials (DFMs) for a reactive carbon capture (RCC) process consisting of simulated direct air capture and subsequent CO2 methanation (DACM). Superior methane desorption purity was observed on TiO2-based DFMs (>94 %) relative to state-of-the-art Al2O3-supported DFMs (77 %). Modifying the process to begin CO2 adsorption immediately following the methanation stage, rather than beginning adsorption near ambient temperature, resulted in >97 % methane desorption purity, suitable for injection into the natural gas pipeline without additional CO2 separation steps.

organic↗
DOE OSTI2020

Open Source Platform for Live Processing of High Data Rate Electron Microscopy (SBIR Phase I Award Topic 15, Subtopic B, Final Scientific / Technical Report)

4D scanning transmission electron microscopy (STEM) is experiencing a revolution both in terms of data size, and data veracity, as are many other areas of experimental data acquisition. This presents enormous opportunities to make new scientific discoveries, but is also hamstrung by aging software that is no longer fit for purpose. Experimental data acquisition platforms must make the same investments in software infrastructure that the simulation community made decades ago to take advantage of high-performance computing (HPC) if they hope to realize the full potential of the latest advances made in detector technology. This requires rethinking the design and deployment of hardware, moving away from small, fixed commodity computing capabilities with closed interfaces to open, modular, scalable systems designed for parallelism and throughput. The technical approach will leverage best-in-class open source technologies, and develop new approaches strategically to build an open source platform for experimental data acquisition targeting high data rate electron microscopy. The project will develop a software platform that uses the power and performance of modern HPC resources to ingest the large volume of data produced by electron microscopes using direct electron detectors. A set of configurable pipelines will be used to enable the processing to be tailored to the problem being investigated with builtin visualization of data at interactive rates. This project will leverage investments made by the DOE and other agencies to provide a powerful software platform for the experimental data community working with most powerful electron microscopes, such as 4D STEM detectors. It will create a platform that serves as a reference for the community. Permissive open source licensing will make the platform available to all, with a simple deployment strategy offering the ability to keep data close to where it is generated. This advanced software platform will require customization for each deployment, creating opportunities for software services that are typically offered to extend open source software platforms.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Transitioning from File-Based HPC Workflows to Streaming Data Pipelines with openPMD and ADIOS2

This paper aims to create a transition path from file-based IO to streaming-based workflows for scientific applications in an HPC environment. By using the openPMP-api, traditional workflows limited by filesystem bottlenecks can be overcome and flexibly extended for in situ analysis. The openPMD-api is a library for the description of scientific data according to the Open Standard for Particle-Mesh Data (openPMD). Its approach towards recent challenges posed by hardware heterogeneity lies in the decoupling of data description in domain sciences, such as plasma physics simulations, from concrete implementations in hardware and IO. The streaming backend is provided by the ADIOS2 framework, developed at Oak Ridge National Laboratory. This paper surveys two openPMD-based loosely-coupled setups to demonstrate flexible applicability and to evaluate performance. In loose coupling, as opposed to tight coupling, two (or more) applications are executed separately, e.g. in individual MPI contexts, yet cooperate by exchanging data. This way, a streaming-based workflow allows for standalone codes instead of tightly-coupled plugins, using a unified streaming-aware API and leveraging high-speed communication infrastructure available in modern compute clusters for massive data exchange. We determine new challenges in resource allocation and in the need of strategies for a flexible data distribution, demonstrating their influence on efficiency and scaling on the Summit compute system. The presented setups show the potential for a more flexible use of compute resources brought by streaming IO as well as the ability to increase throughput by avoiding filesystem bottlenecks.

Poeschel, Franz↗
DOE OSTI2025

Quality-Controlled Meteorological Data from the Flood Control District of Maricopa County (FCDMC) Network, Phoenix, Arizona (1987-2024)

This dataset contains 15- or 30-minute interval meteorological data from the Flood Control District of Maricopa County (FCDMC), Arizona, USA, covering eight key variables across multiple sensor stations between 1987 and 2024. Each variable is stored as a separate CSV file, containing time-series data that have undergone rigorous quality control (QC) procedures and, where appropriate, short-gap interpolation for consistency. The quality control (QC) pipeline consisted of four sequential tests: (1) a range test to ensure all values fall within physically realistic limits, (2) a step test to identify abrupt and implausible changes between consecutive records, (3) a proximity test that validates flagged values from step test using data from nearby stations and exceedance probability thresholds, and (4) a persistence test to detect and remove periods of unrealistically constant readings. These thresholds were calibrated to Arizona’s environmental conditions and sensor specifications. After QC, short gaps (≤2 hours) were linearly interpolated to ensure consistent temporal resolution, except for wind variables. Due to a major upgrade in FCDMC’s data transmission system, only ALERT-2 protocol data (2016–2024) for wind variables are included; earlier ALERT-1 data were excluded because of irregular sampling and high missing rates. This dataset supports regional climate and infrastructure resilience studies by providing standardized, high-resolution meteorological data for the greater Phoenix metropolitan area.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Future of Water Infrastructure and Innovation Summit

Energy and water systems are interdependent, and the U.S. Department of Energy (DOE) has invested in energy and water for several years, including the Energy-Water Desalination Hub (led by the National Alliance for Water Innovation, NAWI), and research and development (R&D) in resource recovery from wastewater, among other areas. The Advanced Manufacturing Office (AMO) at the U.S. Department of Energy (DOE) held The Future of Water Infrastructure and Innovation Summit to inform the understanding of future opportunities in the water space. The virtual summit was held on October 27 and 28, 2020. The organizers gathered information from a diverse group of relevant water and wastewater stakeholders representing academia, industry, government, nongovernmental organizations, and local/regional utilities. Topics from Day 1 discussions included: desalination, water and wastewater treatment/recovery, produced water, industrial management of water, and hydropower, conveyance, and water systems.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2023

Semi-Automatic Geographic Information System Framework for Creating Photo-Realistic Digital Twin Cities to Support Autonomous Driving Research

Digital twin cities are frequently used in vehicle and traffic simulations to render realistic on-road driving scenarios under various traffic and environmental conditions. These digital twins provide a high-fidelity replica of the physical world (e.g., buildings, roads, infrastructures, traffic) to create three-dimensional (3D) virtual-physical environments to support various emerging vehicle and transportation technologies such as connected and automated vehicles. These virtual environments provide a cost-effective digital proving ground to evaluate, validate, and test emerging technologies that include control algorithms, localization, perception, and sensors. Replicating a real-world traffic scenario in a digital twin using a traditional 3D modeling approach is a time-consuming and labor-intensive effort. Here this paper presents a semi-automated spatial framework to construct realistic 3D digital twin cities to support autonomous driving research using readily available geographic information system (GIS) data and 3D prefabricated (prefab) models. We start with a comprehensive review of geospatial data sources of essential digital entities required in a 3D digital twin city and present an integrated GIS-3D modeling pipeline using customized QGIS/GDAL and Blender scripting in Python. The pipeline outputs are realistic 3D digital twin cities compatible with common vehicle simulation software, such as CARLA and IPG CarMaker. The paper closes with a showcase to demonstrate the quality and usability of a digital twin city created to replicate the Shallowford Road corridor in Chattanooga in both Unity and Unreal engine-based virtual environment. The generated digital twin city can be applied to a hardware-in-the-loop simulation environment with an actual testing vehicle to facilitate autonomous driving research.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2026

iDDS: intelligent distributed dispatch and scheduling for workflow orchestration

The intelligent distributed dispatch and scheduling (iDDS) service is a versatile workflow orchestration system designed for large-scale, distributed scientific computing. iDDS extends traditional workload and data management by integrating data-aware execution, conditional logic, and programmable workflows, enabling automation of complex and dynamic processing pipelines. Originally developed for the ATLAS experiment at the large hadron collider, iDDS has evolved into an experiment-agnostic platform that supports both template-driven workflows and a Function-as-a-Task model for Python-based orchestration. This paper presents the architecture and core components of iDDS, highlighting its scalability, modular message-driven design, and integration with systems such as PanDA and Rucio. We demonstrate its versatility through real-world use cases: fine-grained tape resource optimization for ATLAS, orchestration of large Directed Acyclic Graph (DAG) workflows for the Rubin Observatory, distributed hyperparameter optimization for machine learning applications, active learning for physics analyses, and AI-assisted detector design at the electron–ion collider. By unifying workload scheduling, data movement, and adaptive decision-making, iDDS reduces operational overhead and enables reproducible, high-throughput workflows across heterogeneous infrastructures. We conclude with current challenges and future directions, including interactive, cloud-native, and serverless workflow support.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

First Plant Cell Atlas symposium report

The Plant Cell Atlas (PCA) community hosted a virtual symposium on December 9 and 10, 2021 on single cell and spatial omics technologies. The conference gathered almost 500 academic, industry, and government leaders to identify the needs and directions of the PCA community and to explore how establishing a data synthesis center would address these needs and accelerate progress. This report details the presentations and discussions focused on the possibility of a data synthesis center for a PCA and the expected impacts of such a center on advancing science and technology globally. Community discussions focused on topics such as data analysis tools and annotation standards; computational expertise and cyber-infrastructure; modes of community organization and engagement; methods for ensuring a broad reach in the PCA community; recruitment, training, and nurturing of new talent; and the overall impact of the PCA initiative. These targeted discussions facilitated dialogue among the participants to gauge whether PCA might be a vehicle for formulating a data synthesis center. The conversations also explored how online tools can be leveraged to help broaden the reach of the PCA (i.e., online contests, virtual networking, and social media stakeholder engagement) and decrease costs of conducting research (e.g., virtual REU opportunities). Major recommendations for the future of the PCA included establishing standards, creating dashboards for easy and intuitive access to data, and engaging with a broad community of stakeholders. The discussions also identified the following as being essential to the PCA's success: identifying homologous cell-type markers and their biocuration, publishing datasets and computational pipelines, utilizing online tools for communication (such as Slack), and user-friendly data visualization and data sharing. In conclusion, the development of a data synthesis center will help the PCA community achieve these goals by providing a centralized repository for existing and new data, a platform for sharing tools, and new analytical approaches through collaborative, multidisciplinary efforts. A data synthesis center will help the PCA reach milestones, such as community-supported data evaluation metrics, accelerating plant research necessary for human and environmental health.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

GOOML - Finding Optimization Opportunities for Geothermal Operations: Preprint

Geothermal Operational Optimization with Machine Learning (GOOML) is a transferable and extensible component-based geothermal asset modeling framework that considers complex steamfield relationships and identifies optimization prospects using a data-driven approach. We have used this framework to develop digital twins that provide steamfield operators with an operational environment to analyze and understand historical and forecasted power production, explore new steamfield configuration possibilities, and seek optimal asset management for real world applications. The GOOML modeling software is built on a generic component-based systems framework that allows for both historical and forecast analysis. A GOOML model can perform historical data-assimilation using first-principal thermodynamics to create a meaningful data model. Historical production data can then be coupled with a forecast framework to train machine-learning models of steamfield components to predict future outputs. This modeling environment enables digital exploration of steamfield design configurations and operational scenarios. GOOML digital twins have been developed for steamfields in New Zealand and the United States representing differing power generation and field conditions. These digital twins have been validated by comparing hindcast predictions against historical production data. Reinforcement learning experiments were conducted to demonstrate the ability to programmatically explore the operations space using machine learning agents. Our initial results are compelling; two to five percent increases in annual energy production were demonstrated by the GOOML models with no additional infrastructure build required. GOOML offers a new approach to geothermal operations by applying state-of-the-art machine learning algorithms, comprehensive data analytics, and interaction with digital twins. Through application of these tools, operators will realize greater availability and higher net generation which will increase the cost effectiveness of geothermal energy projects.

access↗
DOE OSTI2022

Design of a Novel Information System for Semi-Automated Management of Cybersecurity in Industrial Control Systems

There is an urgent need in many critical infrastructure sectors, including the energy sector, for attaining detailed insights into cybersecurity features and compliance with cybersecurity requirements related to their Operational Technology (OT) deployments. Frequent feature changes of OT devices interfere with this need, posing a great risk to customers. One effective way to address this challenge is via a semi-automated cyber-physical security assurance approach, which enables verification and validation of the OT device cybersecurity claims against actual capabilities, both pre- and post-deployment. To realize this approach, this paper presents new methodology and algorithms to automatically identify cybersecurity-related claims expressed in natural language form in ICS device documents. Here, we developed an identification process that employs natural language processing (NLP) techniques with the goal of semi-automated vetting of detected claims against their device implementation. We also present our novel NLP components for verifying feature claims against relevant cybersecurity requirements. The verification pipeline includes components such as automated vendor identification, device document curation, feature claim identification utilizing sentiment analysis for conflict resolution, and reporting of features that are claimed to be supported or indicated as unsupported. Our novel matching engine represents the first automated information system available in the cybersecurity domain that directly aids the generation of ICS compliance reports.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2025

Toward a persistent event-streaming system for high-performance computing applications

High-performance computing (HPC) applications have traditionally relied on parallel file systems and file transfer services to manage data movement and storage. Alternative approaches have been proposed that use direct communications between application components, trading persistence and fault tolerance for speed. Event-driven architectures, as popularized in enterprise contexts, present a compelling middle ground, avoiding the performance cost and API constraints of parallel file systems while retaining persistence and offering impedance matching between application components. However, adapting streaming frameworks to HPC workloads requires addressing challenges unique to HPC systems. This paper investigates the potential for a streaming framework designed for HPC infrastructures and use cases. We introduce Mofka, a persistent event-streaming framework designed specifically for HPC environments. Mofka combines the capabilities of a traditional streaming service with optimizations tailored to the HPC context, such as support for massively multicore nodes, efficient scaling for large producer-consumer workflows, RDMA-enabled high-performance network communications, specialized network fabrics with multiple links per node, and efficient handling of large scientific data payloads. Built using the Mochi suite of HPC data service components, Mofka provides a lightweight, modular, and high-performance solution for persistent streaming in HPC systems. We present the architecture of Mofka and evaluate its performance against Kafka and Redpanda using benchmarks on diverse platforms, including Argonne's Polaris and Oak Ridge's Frontier supercomputers, showing up to 8× improvement in throughput in some scenarios. We then demonstrate its utility in several real-world applications: a tomographic reconstruction pipeline, a workflow for the discovery of metal-organic frameworks for carbon capture, and the instrumentation of Dask workflows for provenance tracking and performance analysis.

HPC↗
DOE OSTI2022

Optimal Gas-Electric Energy System Decarbonization Planning

As energy utilities implement climate change mitigation policies, system planners require strategies for achieving affordable emissions reductions. Coordinated planning of electric power and natural gas systems will allow synergistic investments to address cross-sector operational constraints, competing uses for net-zero emissions fuels, and shifts in energy demands across energy carriers. In this study, we develop a novel optimization program that finds the cost-minimizing mix of infrastructure expansion or reduction across gas and electric systems to satisfy sector-specific emissions constraints. Alongside energy supply resources, our framework allows for central-planning of end-use equipment stocks to allow switching between gas and electric appliances upon failure or premature replacement. The proposed model is used to simulate case study scenarios for a benchmark 24-pipe gas network coupled to a 24-node power system test network. We find that electrification of greater than 80% of core gas demands is a component of the least-cost solution for modeled energy systems. Despite this substitution, the gas system is maintained to service difficult-to-electrify customers and to deliver net-zero emissions gas to electricity generators in times of peak electricity demand. Restricting electrification of gas appliances increases reliance on power- to-gas technologies and increases annual costs by 15% in 2040. Neglecting constraints on pipeline blending of hydrogen can produce a misleading result that relies on hydrogen blend fractions of greater than 50%. In all cases, we find the average costs of delivered gas increase nearly 5-fold across the decarbonization transition, highlighting the importance of future work investigating cost-allocation strategies for ensuring an equitable energy transition.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗