SEARCH · Search NASA

Results for “End-to-end simulations”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

All sources NASA NTRS DOE OSTI

DOE OSTI2026

Detector Interface for Streaming, Control, and Open-source integration (DISCO) v1.0.0

This suite consists of a multi-package ecosystem featuring detector emulators, EPICS areaDetector drivers, and remote server frameworks designed for the Advanced Light Source (ALS). Engineered for high-bandwidth devices—including VFCCD, Timepix3, Timepix4, and related pixel detectors—the software simulates hardware, wraps vendor SDKs into remote-callable servers, and integrates with open-source control systems. Key Capabilities: Distributed SDK Architecture: Server packages wrap hardware-specific SDKs, allowing areaDetector drivers to execute remote framework calls. This isolates proprietary libraries from the EPICS IOC, enhancing stability and enabling distributed computing across beamline networks. Device Support: Custom drivers for VFCCD, the Timepix family, and similar sensors optimize the data path from hardware control to high-speed transport. Full-Stack Emulation: Sophisticated emulator packages allow end-to-end pipeline testing and software development without requiring physical hardware or beam time. Integrated Workflows: Supports high-bandwidth streaming for real-time analysis and robust, metadata-rich file-based workflows (e.g., HDF5/NeXus). By standardizing interfaces across heterogeneous hardware, this suite reduces technical debt. It provides the ALS with a scalable, open-source solution to manage massive data rates within a unified control environment.

Mahl, Johannes [Lawrence Berkeley National Laborat↗

DOE OSTI2024

Advances in ArborX to support exascale applications

ArborX is a performance portable geometric search library developed as part of the Exascale Computing Project (ECP). In this paper, we explore a collaboration between ArborX and a cosmological simulation code HACC. Large cosmological simulations on exascale platforms encounter a bottleneck due to the in-situ analysis requirements of halo finding, a problem of identifying dense clusters of dark matter (halos). This problem is solved by using a density-based DBSCAN clustering algorithm. With each MPI rank handling hundreds of millions of particles, it is imperative for the DBSCAN implementation to be efficient. In addition, the requirement to support exascale supercomputers from different vendors necessitates performance portability of the algorithm. We describe how this challenge problem guided ArborX development, and enhanced the performance and the scope of the library. We explore the improvements in the basic algorithms for the underlying search index to improve the performance, and describe several implementations of DBSCAN in ArborX. Further, we report the history of the changes in ArborX and their effect on the time to solve a representative benchmark problem, as well as demonstrate the real world impact on production end-to-end cosmology simulations.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

Enabling kilometer-scale E3SM land model simulation over North America: A new integrated framework solution

This study introduces a novel framework designed to enhance the performance, scalability, and portability of the kilometer-scale E3SM Land Model (km-ELM) within the E3SM modeling infrastructure. By seamlessly integrating cutting-edge data tools, we address existing challenges such as slow performance, limited scalability, and difficulties in software integration in current data-driven ELM simulation over large geographic areas. Our innovative approach leverages the KiloCraft data toolkit to generate unified inputs for simulations ranging from a single-cite case, to a 72,083-cell regional case to a continental configuration encompassing 21.6 million land grid cells at a 1 km × 1 km resolution. We conduct extensive strong- and weak-scaling experiments on three state-of-the-art supercomputers, utilizing up to 100,800 CPU cores across 2400 compute nodes to evaluate end-to-end metrics including wall-clock time, simulation-years-per-day (SYPD), initialization costs, and I/O throughput. Our results reveal the land (LND) component’s efficient scaling, demonstrating near-ideal weak scaling and strong-scaling parallel efficiencies reaching up to 87% at 50,400 cores. We confirm portability and reproducibility through bitwise-equivalent outputs across different machines using identical inputs over supported machines. Notably, at extreme scales, we identify I/O as a critical bottleneck and that leads to effective solution with the SCORPIO/ADIOS stack. Collectively, these findings validate the deployment of km-ELM at a continental scale with high parallel efficiency and provide essential guidance on configuration, decomposition, and I/O settings for optimized kilometer-scale land simulations in E3SM. This work emphasizes the innovative design and practical solutions that enhance the operational capabilities of km-ELM, focusing on software performance and scalability while leaving detailed scientific evaluations of simulated land processes for future investigations.

E3SM land model (ELM), km-ELM, scalability, perfor↗

DOE OSTI2026

From 2D to 4D: a containerized workflow and browser to explore dynamic chromatin architecture

Background Characterizing the physical organization of the genome is essential for understanding long-range gene regulation, chromatin compartmentalization, and epigenetic accessibility. Hi-C experiments generate two-dimensional (2D) genome-wide contact maps of chromatin interactions by capturing the spatial proximity between genomic loci, which reveal interaction frequencies but lack the spatial resolution needed to interpret the three-dimensional (3D) genome structure(s). Emerging evidence suggests that epigenetic regulation is closely linked to 3D genome architecture, and that structural changes over time (4D) drive key biological processes in development, disease, and environmental response. Thus, integrating 3D structure with functional data is critical for a more complete understanding of genome regulation. Previous work, most notably the 4DHiC chromosome modeling framework, has shown that physical multi-dimensional modeling approaches rooted in polymer physics and molecular dynamics can resolve these structures at biologically meaningful resolutions by integrating temporal Hi-C data with physical constraints to uncover dynamic chromosome reorganization. Thus, molecular dynamics simulations, constrained by Hi-C contact matrices, can resolve fine-scale structural changes and reveal functionally significant transitions in chromatin conformation. Results Herein, we present the 4D Genome Browser Workflow (4DGBWorkflow) and the 4D Genome Browser (4DGB). The algorithm is based on the 4DHiC method, and the containerized tool is an end-to-end workflow that can transform, filter, and view 4D epigenomics and chromatin datasets, allowing non-specialists to apply three-dimensional modeling principles to diverse datasets and experimental conditions. The software executes on a laptop running macOS, Linux or Windows. From input Hi-C files (.hic), the 4DGBWorkflow produces 3D reconstructions of chromosomes, integrates the reconstruction with track data (e.g., epigenetic marks, transcriptome profiles), and provides comparative visualization of the results in a single workflow. Conclusions The 4DGBWorkflow and 4D Genome Browser are open-source tools for comparative analysis and visualization of 4D chromosome datasets, including chromatin architecture and epigenomic signals. Automatic integration of Hi-C data with molecular dynamics democratizes the construction of time resolved 3D genome structures, simplifying complex simulations and data integration schemes.

3D Genome Browser↗

DOE OSTI2026

Data for An End-to-End Pipeline for Succinic Acid Production at an Industrially Relevant Scale Using Issatchenkia orientalis

Microbial production of succinic acid (SA) at an industrially relevant scale has been hindered by high downstream processing costs arising from neutral pH fermentation for over three decades. Here, we metabolically engineer the acid-tolerant yeast Issatchenkia orientalis for SA production, attaining the highest titers in sugar-based media at low pH (pH 3) in fed-batch fermentations, i.e. 109.5 g/L in minimal medium and 104.6 g/L in sugarcane juice medium. We further perform batch fermentation using sugarcane juice medium in a pilot-scale fermenter (300×) and achieve 63.1 g/L of SA, which can be directly crystallized with a yield of 64.0%. Finally, we simulate an end-to-end low-pH SA production pipeline, and techno-economic analysis and life cycle assessment indicate our process is financially viable and can reduce greenhouse gas emissions by 34–90% relative to fossil-based production processes. We expect I. orientalis can serve as a general industrial platform for production of organic acids.

Metabolomics↗

DOE OSTI2020

An intra-node HPC network architecture with nanosecond-scale photonic switches

We propose a single-stage network architecture for intra-node connectivity that makes use of nanosecond-scale photonic switches. Although buffering at the switch points is of vital importance for complex multi-stage networks, this is not the case for smaller-scale single-stage networks where the end nodes are located only one hop apart. By limiting the buffering to the end points, the proposed architecture manages to minimize the required electro-optic and opto-electronic conversions, leading in this way to both low end-to-end latency and better energy efficiency. Combining these advantages with nanosecond-scale switching times can allow for high-throughput operation even for frequent switch reconfigurations. The performance of the proposed architecture is evaluated via discrete-event simulations for a wide range of synthetic-traffic cases. The simulation results show that high-throughput operation of ≥90% can be achieved even for small message sizes, i.e., 32 KB for all-to-all communication and 2 KB for uniform random traffic, at a data rate of 400 Gb/s and a switch reconfiguration time of ≤72 ns. Moreover, if 100-ns reconfiguration times are achievable as opposed to 150-ns, then for the all-to-all traffic case a 16% and 28% reduction in completion time can be achieved for message sizes of 8 KB and 1 KB, respectively. In a forthcoming era of optically interfaced processors and accelerators, nanosecond-scale photonic switches appear as a highly promising solution for keeping up with the intra-node bandwidth scaling due to their high-bandwidth, low-latency and fast-switching capabilities.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2021

Machine-learning-aided cognitive reconfiguration for flexible-bandwidth HPC and data center networks [Invited]

This paper proposes a machine-learning (ML)-aided cognitive approach for effective bandwidth reconfiguration in optically interconnected datacenter/high-performance computing (HPC) systems. The proposed approach relies on a Hyper-X-like architecture augmented with flexible-bandwidth photonic interconnections at large scales using a hierarchical intra/inter-POD photonic switching layout. We first formulate the problem of the connectivity graph and routing scheme optimization as a mixed-integer linear programming model. A two-phase heuristic algorithm and a joint optimization approach are devised to solve the problem with low time complexity. Then, we propose an ML-based end-to-end performance estimator design to assist the network control plane with intelligent decision making for bandwidth reconfiguration. Numerical simulations using traffic distribution profiles extracted from HPC applications traces as well as random traffic matrices verify the accuracy performance of the ML design estimator ( < <#comment/> 9 % <#comment/> error) and demonstrate up to 5 × <#comment/> throughput gain from the proposed approach compared with the baseline Hyper-X network using fixed all-to-all intra/inter-portable data center interconnects.

Chen, Xiaoliang (ORCID:0000000278056237)↗

DOE OSTI2022

Sim2Ls: FAIR simulation workflows and data

Just like the scientific data they generate, simulation workflows for research should be findable, accessible, interoperable, and reusable (FAIR). However, while significant progress has been made towards FAIR data, the majority of science and engineering workflows used in research remain poorly documented and often unavailable, involving ad hoc scripts and manual steps, hindering reproducibility and stifling progress. We introduce Sim2Ls (pronounced simtools) and the Sim2L Python library that allow developers to create and share end-to-end computational workflows with well-defined and verified inputs and outputs. The Sim2L library makes Sim2Ls , their requirements, and their services discoverable, verifies inputs and outputs, and automatically stores results in a globally-accessible simulation cache and results database. This simulation ecosystem is available in nanoHUB, an open platform that also provides publication services for Sim2Ls , a computational environment for developers and users, and the hardware to execute runs and store results at no cost. We exemplify the use of Sim2Ls using two applications and discuss best practices towards FAIR simulation workflows and associated data.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2025

APOLLO: a facility-scale differentiable virtual accelerator at Fermilab FAST/IOTA

As the design complexity of modern accelerators grows, there is more interest in using advanced simulations that have fast execution time or yield additional insights like gradients. The FAST/IOTA facility has been working on implementing and experimentally validating an end-to-end digital twin that is both fast and gradient-aware, allowing for rapid prototyping of new software and experiments with minimal beam time costs. Our framework integrates physics and ML codes for linac and ring simulation through a set of generic interfaces between surrogate and physics-based sections. To reproduce device inputs and outputs, system state is exposed as a deterministic event loop in a specialized discrete event simulator architecture. Because Fermilab is undergoing control system transition, several APIs were implemented as final user interfaces - a fully asynchronous EPICS soft IOC, a gRPC-based Data Pool Manager (DPM), and legacy ACNET protocols. We discuss implementation details as well as challenges handling live data assimilation and future plans to extend modelling to main complex proton accelerators like PIPII and Booster.

Kuklev, Nikita [Fermilab]↗

DOE OSTI2021

`SkyPy`: A package for modelling the Universe

SkyPy is an open-source Python package for simulating the astrophysical sky. It comprises a library of physical and empirical models across a range of observables and a command-line script to run end-to-end simulations. The library provides functions that sample realisations of sources and their associated properties from probability distributions. Simulation pipelines are constructed from these models using a YAML-based configuration syntax, while task scheduling and data dependencies are handled internally and the modular design allows users to interface with external software. SkyPy is developed and maintained by a diverse community of domain experts with a focus on software sustainability and interoperability. By fostering development, it provides a framework for correlated simulations of a range of cosmological probes including galaxy populations, large scale structure, the cosmic microwave background, supernovae and gravitational waves. Version 0.4 implements functions that model various properties of galaxies including luminosity functions, redshift distributions and optical photometry from spectral energy distribution templates. Future releases will provide additional modules, for example, to simulate populations of dark matter halos and model the galaxy-halo connection, making use of existing software packages from the astrophysics community where appropriate.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2025

Virtual Engineering: Python framework for engineering process design

Virtual Engineering (VE) is a Python software framework designed to accelerate the research and development of engineering processes that are fundamentally defined by multiple unit operations executed in series. VE supports a wide variety of different multi-physics models and integrates them to simulate a complete end-to-end process. To automate the execution of this model sequence, VE provides (i) a robust method to communicate between models, (ii) a high-level, user-friendly interface to set model parameters and enable optimization, and (iii) an overall model-agnostic approach that allows new computational units to be swapped in and out of workflows. Although the VE framework was developed to support the biochemical conversion of biomass to fuel, we have designed each component to easily accommodate new domains and unit models.

09 BIOMASS FUELS↗

DOE OSTI2023

Making digital objects FAIR in high energy physics: An implementation for Universal FeynRules Output (UFO) models

Research in the data-intensive discipline of high energy physics (HEP) often relies on domain-specific digital contents. Reproducibility of research relies on proper preservation of these digital objects. This paper reflects on the interpretation of principles of Findability, Accessibility, Interoperability, and Reusability (FAIR) in such context and demonstrates its implementation by describing the development of an end-to-end support infrastructure for preserving and accessing Universal FeynRules Output (UFO) models guided by the FAIR principles. UFO models are custom-made python libraries used by the HEP community for Monte Carlo simulation of collider physics events. Our framework provides simple but robust tools to preserve and access the UFO models and corresponding metadata in accordance with the FAIR principles.

Neubauer, Mark S.↗

DOE OSTI2023

Codebase release 2.0 for UFOManager

Neubauer, Mark S.↗

DOE OSTI2024

Reduce, reuse, reinterpret: An end-to-end pipeline for recycling particle physics results

Searches for new physics at the Large Hadron Collider have constrained many models of physics beyond the Standard Model. Many searches also provide resources that allow them to be reinterpreted in the context of other models. We describe a reinterpretation pipeline that examines previously untested models of new physics using supplementary information from ATLAS Supersymmetry (SUSY) searches in a way that provides accurate constraints even for models that differ meaningfully from the benchmark models of the original analysis. The public analysis information, such as public analysis routines and serialized probability models, is combined with common event generation and simulation toolkits MADGRAPH, PYTHIA8, and DELPHES into workflows steered by TOML configuration files, and bundled into the mapyde Python package. The use of mapyde is demonstrated by constraining previously untested SUSY models with compressed sleptons and electroweakinos using ATLAS results.

Stark, Giordon↗

DOE OSTI2020

Predictive Modeling and Diagnostic Monitoring of Extreme Science Workflows (Final Report)

This proposal addresses a critical issue of performance prediction identified in the report from the ASCR \Computational Modeling of Big Networks (COMBINE)" workshop: "end-to-end performance is not predictable due to a variety of factors. Even when some performance forecasts or predictions can be made, they often cannot explain the reasons why some predictions fail." We will develop new analytical models to predict the end-to-end performance of scientific workflows on DOE computing infrastructures, and use simulations and experimentation to validate and refine these models, as well as to pinpoint the sources of model inaccuracy. We will also use these models to help diagnose application and infrastructure problems, and to adapt the system based on this diagnosis. This section provides background in the areas relevant to the proposed work. RPI’s specific tasks within the Panorama project are as follows: (1) Develop Aspen-Simulation interface for Workflow Model Driven Simulation. (2) Validate manual performance models of two target workflow scenarios with empirical measurement and simulation. (3) Extend ROSS-Aspen API to simulate workflow descriptions when required. (4) Validate Aspen performance models of two target workflow scenarios with automatic performance model empirical measurement and simulation. (5) Design and implement final system to automatically generate Aspen performance models from workflow descriptions (including methods to compensate for limitations of Aspen analytical models). (6) Validate improved Aspen performance models with target workflow on production infrastructure. To date, all the project milestones assigned to us where reached within the best of our abilities over the course of the project performance period. Below describes the key outcome from our collaborative research in a system named, Durango .

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Noisy Intermediate-Scale Quantum Applications on a Pathfinder System

Work performed under this one-year LDRD was concerned with estimating resource requirements for small quantum test beds that are expected to be available in the near future. This work represents a preliminary demonstration of our ability to leverage quantum hardware for solving small quantum simulation problems in areas of interest to the DOE. The algorithms enabling such studies are hybrid quantum-classical variational algorithms, in particular the widely-used variational quantum eigensolver (VQE). Employing this hybrid algorithm, in which the quantum computer complements the classical one, we implemented an end-to-end application-level toolchain that allows the user to specify a molecule of interest and compute the ground state energy using the VQE approach. We found significant limitations attributable to the classical portion of the hybrid system, including a greater than greater-than-quartic power scaling of the classical memory requirements with the system size. Current VQE approaches would require an exascale machine for solving any molecule with size greater than 150 nuclei. Our findings include several improvements that we implemented into the VQE toolchain, including a new classical optimizer that is decades old but hadn't been considered before in the VQE ecosystem. Our findings suggest limitations to variational hybrid approaches to simulation that further motivate the need for a gate-based fault-tolerant quantum processor that can implement larger problems using the fully digital quantum phase estimation algorithm.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2020

Advanced High-Performance Computational Modeling of the Seismic Response of High-Hazard and/or Nuclear Facilities and Critical Infrastructure at the NNSS

New methods for predicting the amplitude and variability of ground shaking from earthquakes (and explosions) are needed for seismic hazard analysis for buildings, nuclear power plants, and critical infrastructure at the NNSS. We are comparing existing 1-D and new 3-D geophysical methods for estimating the shear-wave velocity structure in the upper 30 meters of the ground surface (Vs30), which plays a major role in ground motion amplification and seismic response of buildings. We evaluate the performance of these methodologies at the U1a Complex at the NNSS and develop simple 1-D and high-resolution 3-D Vs30 models. We then emplace these high-resolution models into a background seismic velocity model. We will collaborate with Lawrence Livermore National Laboratory (LLNL) to conduct numerical modeling of the ground shaking at the NNSS using their high-performance computing technology and state-of-the-art ground motion simulation methodology. The primary work that was completed in FY 2019 was to acquire the seismic systems and familiarize staff at the NNSS with their use. We also worked on developing a collection plan with the Device Assembly Facility (DAF) at the NNSS, but due to time constraints and other ongoing projects at the DAF, we had to use U1a as a backup. We were able to coordinate the seismic survey, and we will complete the collection of seismic data in FY 2020. Additionally, during FY 2019, we completed the geologic framework model (GFM) for the U1a Complex and modeled the Yucca fault. LLNL worked with us through FY 2019 to prepare the data files for their modeling software and tested the software for reliability. In FY 2020 we will develop the end-to-end capability so that any facility could easily be modeled and the expected shaking from a local earthquake understood. The work in FY 2020 will include building fault models from the GFM and finalizing the velocity model analysis. The final simulations will be run for multiple rupture models, and final assessments will demonstrate the seismic hazard at the U1a Complex.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2021

End-to-End Differentiable Modeling and Management of the Environment

Focal Area: (2) Data acquisition and assimilation enabled by machine learning, AI, and advanced methods including experimental/network design/optimization. We emphasize the importance of leveraging optimization techniques from AI/machine learning (ML) to solve challenging problems in Earth system modeling. Science Challenge: Automatic differentiation has had a transformative effect on ML by allowing the calculation of gradients of arbitrary functions in an incredibly large class of models. We can potentially realize similar improvements in parameter estimation and control for Earth system models (ESMs) by reimplementing them in computational frameworks from ML. Practitioners working with large (>10 7 parameters) models in ML and AI can obtain good predictive performance in a range of spatiotemporal tasks by making use of optimization via stochastic gradient descent and incorporating prior knowledge at multiple levels. We propose writing ESMs in open-source computational frameworks such as Torch, Tensorflow, and JAX to greatly expand the scope of environmental forecasting and management challenges, which can be addressed by leveraging automatic differentiation and gradient descent-like algorithms. We do not call for a wholesale replacement of physical models with data-driven surrogates, but rather advocate for interleaving physical and empirical equations in a manner that is most faithful to the extent of our scientific knowledge and observational data. Central to this topic is the merging of differentiable physical simulations with differentiable optimization layers, which are now both beginning to come to the forefront.

54 ENVIRONMENTAL SCIENCES↗