Search NASA⌕ Search

SEARCH · Search NASA

Results for “data transfer nodes”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

62 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Toward a persistent event-streaming system for high-performance computing applications

High-performance computing (HPC) applications have traditionally relied on parallel file systems and file transfer services to manage data movement and storage. Alternative approaches have been proposed that use direct communications between application components, trading persistence and fault tolerance for speed. Event-driven architectures, as popularized in enterprise contexts, present a compelling middle ground, avoiding the performance cost and API constraints of parallel file systems while retaining persistence and offering impedance matching between application components. However, adapting streaming frameworks to HPC workloads requires addressing challenges unique to HPC systems. This paper investigates the potential for a streaming framework designed for HPC infrastructures and use cases. We introduce Mofka, a persistent event-streaming framework designed specifically for HPC environments. Mofka combines the capabilities of a traditional streaming service with optimizations tailored to the HPC context, such as support for massively multicore nodes, efficient scaling for large producer-consumer workflows, RDMA-enabled high-performance network communications, specialized network fabrics with multiple links per node, and efficient handling of large scientific data payloads. Built using the Mochi suite of HPC data service components, Mofka provides a lightweight, modular, and high-performance solution for persistent streaming in HPC systems. We present the architecture of Mofka and evaluate its performance against Kafka and Redpanda using benchmarks on diverse platforms, including Argonne's Polaris and Oak Ridge's Frontier supercomputers, showing up to 8× improvement in throughput in some scenarios. We then demonstrate its utility in several real-world applications: a tomographic reconstruction pipeline, a workflow for the discovery of metal-organic frameworks for carbon capture, and the instrumentation of Dask workflows for provenance tracking and performance analysis.

HPC↗
DOE OSTI2020

Accurate Atomic Correlation and Total Energies for Correlation Consistent Effective Core Potentials

Very recently, we introduced a set of correlation consistent effective core potentials (ccECPs) constructed within full many-body approaches. By employing significantly more accurate correlated approaches, we were able to reach a new level of accuracy for the resulting effective core Hamiltonians. We also strived for simplicity of use and easy transferability into a variety of electronic structure methods in quantum chemistry and condensed matter physics. Here, as a reference for future use, we present exact or nearly exact total energy calculations for these ccECPs. The calculations cover H–Kr elements and are based on the state-of-the-art configuration interaction (CI), coupled-cluster (CC), and quantum Monte Carlo (QMC) calculations with systematically eliminated/improved errors. In particular, we carry out full CI/CCSD(T)/CCSDT(Q) calculations with cc-pVnZ with up to n = 6 basis sets and we estimate the complete basis set limits. Using combinations of these approaches, we achieved an accuracy of ≈1–10 mHa for K–Zn atoms and ≈0.1–0.3 mHa for all other elements—within about 1% or better of the ccECP total correlation energies. We also estimate the corresponding kinetic energies within the feasible limit of full CI calculations. In order to provide data for QMC calculations, we include fixed-node diffusion Monte Carlo energies for each element that give quantitative insights into the fixed-node biases for single-reference trial wave functions. The results offer a clear benchmark for future high-accuracy calculations in a broad variety of correlated wave function methods such as CI and CC as well is in stochastic approaches such as real space sampling QMC.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2026

Stakeholder-guided holistic, Adaptive Framework for enhancing community Energy Resilience (SAFER) (Final Technical Report)

The Stakeholder-guided holistic, Adaptive Framework for enhancing community Energy Resilience (SAFER) project advances resilience science and engineering by addressing challenges in rural Kansas communities where aging infrastructure, extreme weather, and socioeconomic disparities heighten vulnerability to energy disruptions. Traditional approaches often focus on technical performance while overlooking community concerns and priorities. SAFER responds by integrating community perspectives with advanced analytical frameworks to create a holistic model for measuring and improving resilience. Project objectives included developing novel resilience metrics, advancing modeling frameworks that capture interdependencies across infrastructures, and embedding community-centric indicators directly into planning processes for distributed energy resources. The key technical innovations included the creation of self-organizing map (SOM)-based indices for objective resilience quantification, hetero-functional graph theory (HFGT) models linking power, water, transportation, and community assets, and graph neural network (GNN) tools for identifying critical nodes in complex systems. Community-centric energy planning was demonstrated through optimal siting and sizing of (photovoltaic) PV and battery storage, ensuring resilience enhancements also addressed energy burden and energy insecurity. SAFER engaged community partners in Dodge City and Ford County through surveys, focus groups, and workshops, generating more than 600 responses that established baseline measures of energy burden, financial insecurity, and willingness-to-pay to avoid outages. This data, organized in terms of a community capitals framework, informed the development of weighted reliability indices that better reflect community costs than traditional utility metrics. SAFER’s GNN-based critical node identification framework identified expert-labelled critical nodes with over 99% accuracy, while also uncovering additional functionalities essential for proactive resilience planning. The project’s models demonstrated that optimal PV and storage deployment could improve resilience indices by over 11 percent, with dispatch strategies further enhancing outcomes, confirming both the technical effectiveness and economic feasibility of these approaches. Through its combined emphasis on rigorous modeling, community-focused planning, and community engagement, SAFER advances the state of resilience research while delivering direct benefits to rural communities. The project provides tools, guidelines, and resilience heatmaps that help utilities, local governments, and residents better anticipate disruptions, prioritize investments, and strengthen the capacity to withstand and recover from energy-related hazards. Furthermore, the developed HFG and GNN frameworks are designed for transferability, allowing them to be adapted for resilience planning in other communities with minimal retraining. This inductive learning capability provides a scalable pathway to extend the SAFER project’s impact. Thus, creating a foundation for a nationally applicable model of infrastructure resilience. Additionally, the HFG can also be extended to include other FEMA community lifelines.

14 SOLAR ENERGY↗
DOE OSTI2021

Real Time - Optimal Power Flow Based Distributed Energy Resources Management System (DERMS) (CRADA Number CRD-20-16909 Final Report)

The integration of behind-the-meter distributed energy resources (DERs) into distribution systems brings transformative changes to power systems. This requires operators and planners to find solutions to modernize electric grids and to effectively manage DERs for grid services. NREL developed novel DER management algorithms (referred to as Real-Time Optimal Power Flow, RT-OPF) through U.S. Department of Energy (DOE)-funded efforts, including Advanced Research Projects Agency–Energy (ARPA-E) Network Optimized Distributed Energy Systems (NODES) funding. This cutting-edge control technology aims to modernize distribution systems with large amounts of DER integration, which will help utilities solve issues brought by renewable integration and build resilient and renewable-based electric grids nationwide. Utilidata worked with NREL to investigate the commercialization opportunity of this RT-OPF-based distributed energy resource management system (DERMS). In this project, NREL performs the technology transfer of the RT-OPF to Utilidata to help them fully understand the RT-OPF solution, to identify potential engineering hurdles, and to assess the expected commercial value of various RT-OPF use cases and deployment. The technology transfer work includes two major tasks. First, NREL performs an in-depth knowledge transfer of the entire RT-OPF solution to Utilidata to help them gain an extensive and detailed understanding of the complete RT-OPF solution. In this task, NREL provides exhaustive information (e.g., documentation, code packages, laboratory and field trial data, performance results) while conducting in-depth training sessions to provide a thorough explanation of the entire solution. NREL hosts meetings to present different topics related to the RT-OPF solution, and question-and-answer sessions are included in each meeting to better explain the RT-OPF-related work. Second, the RT-OPF simulations are performed in a laboratory environment. The main objectives of this task are to walk through with Utilidata engineers how to set up a simulation of RT-OPF, identifying each RT-OPF code block/component in operation, learning how these components interact with each other, and eventually running RT-OPF simulations under various system conditions. This task helps Utilidata engineers understand performance limitations and constraints while also quantifying the commercial value of the RT-OPF for different use cases. Based on these two tasks, Utilidata engineers should be able to define and prioritize the next steps of RT-OPF implementation with an eye toward commercial success and scalability of the solution. The next steps are expected to be part of a new project following the conclusion of this project.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

Region-adaptive, Error-controlled Scientific Data Compression using Multilevel Decomposition

The increase of computer processing speed is significantly outpacing improvements in network and storage bandwidth, leading to the big data challenge in modern science, where scientific applications can quickly generate much more data than that can be transferred and stored. As a result, big scientific data must be reduced by a few orders of magnitude while the accuracy of the reduced data needs to be guaranteed for further scientific explorations. Moreover, scientists are often interested in some specific spatial/temporal regions in their data, where higher accuracy is required. The locations of the regions requiring high accuracy can sometimes be prescribed based on application knowledge, while other times they must be estimated based on general spatial/temporal variation. In this paper, we develop a novel multilevel approach which allows users to impose region-wise compression error bounds. Our method utilizes the byproduct of a multilevel compressor to detect regions where details are rich and we provide the theoretical underpinning for region-wise error control. With spatially varying precision preservation, our approach can achieve significantly higher compression ratios than single-error bounded compression approaches and control errors in the regions of interest.We conduct the evaluations on two climate use cases – one targeting small-scale, node features and the other focusing on long, areal features. For both use cases, the locations of the features were unknown ahead of the compression. By selecting approximately 16% of the data based on multi-scale spatial variations and compressing those regions with smaller error tolerances than the rest, our approach improves the accuracy of post-analysis by approximately 2 × compared to single-error-bounded compression at the same compression ratio. Using the same error bound for the region of interest, our approach can achieve an increase of more than 50% in overall compression ratio.

Gong, Qian↗
DOE OSTI2022

SMALE: Enhancing Scalability of Machine Learning Algorithms on Extreme-Scale Computing Platforms

Deployment and execution of machine learning tasks on extreme-scale computing platforms face several significant technical challenges: 1) High computing cost incurred by dense networks – The computing workload of deep networks with densely-connected topology increases rapidly with the network size, imposing a non-scalable computing model of extreme-scale computing platforms; 2) Non-optimized workload distribution – Many advanced deep learning algorithms, e.g., sparsification and irregular net-work topology, produce very unbalanced workload distribution on extreme-scale computing platforms. The computation efficiency is greatly hindered by the incurred data and computation redundancies as well as long tails of the node with extensive workload; 3) Constraints in data movement and I/O bottle-neck – Inter-node data movement in extreme-scale computing platforms are associated with high energy and latency costs, and subject to the constraints of I/O bandwidth; and 4) Generalization of algorithm realization and acceleration on computing platforms – The large varieties of machine learning algorithms and structures of extreme-scale computing platforms make the derivation of a generalized algorithm realization and acceleration method very challenging, which, however, is the requirement by domain scientists and interested users. We call the above challenges Smale’s Problems in Machine Learning and Understanding for High-Performance Computing Scientific Discovery. The objective of our three-year research project is to develop a holistic innovation set at structure, assembly, and acceleration layers of machine learning algorithms to address the above challenges in algorithm deployment and execution. Three tasks are particularly performed, including: At the algorithm structure level, we investigate the techniques that can structurally sparsify on the topology of deep networks for computing workload reduction. We also study clustering and pruning techniques that can optimize the workload distributions over the extreme-scale computing platforms; At the algorithm assembly level, we derive a unified learning framework for unsupervised transfer learning and dynamic growing capabilities. Novel training methods are also exploited to enhance the training efficiency of the proposed framework; At the algorithm acceleration level, we will develop a series of techniques that can accelerate the computation of sparse matrix operations, which are one of the core executions in deep learning and optimize memory access of the concerned platforms. Our proposed techniques attack the fundamental problems in machine learning algorithms running on extreme-scale computing platforms by vertically integrating the solutions at three closely entangled layers, paving the long-term scaling path of machine learning applications under DOE context. Three tasks corresponding to the above respective research orientations are performed during the three-year project period with our collaborators at ORNL. The outcome of the proposed project is anticipated to form a holistic solution set of novel algorithms and network topologies, efficient training techniques, and fast acceleration methods to promote the computing scalability of the machine learning applications of particular interest to DOE.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

LLNL Kimberlina 1.2 NUFT Simulations June 2018 (v2)

This dataset contains the output 6,000, 3-dimensional reactive multi-phase flow and transport aquifer simulations of brine and CO2 leakage into a protective aquiver in California’s San Joaquin Valley and input data files detailing the geologic mesh, aquifer physical properties and CO2 and brine injection rates. This data set was generated as an ongoing effort with the US DOE National Risk Assessment Partnership (NRAP) to evaluate the effectiveness of monitoring techniques to detect brine and CO2 leakage from legacy wells into underground sources of drinking water overlaying a CO2 storage reservoir. Each simulation contains a unique set of input parameters, generated stochastically. The outputs consist of these upper three geologic layers (from top): the Etchegoin, Macoma-Chanac, Santa Margarita-McLure formations. These simulations span the several distances (1, 3 and 6 km or wells W31-0.2, W31-0.5 and W31-1.0, respectively) from the CO2 injector, initiated from bottom hole pressure and saturation to calculate wellbore leakage from the storage reservoir, with low and high regional groundwater gradients and wellbore leakage into 5 leaky nodes. The dataset includes 1,000 unique simulations for each distance, which each contain a unique aquifer heterogeneity, aquifer and caprock permeability, and two model generations are included with a high permeability (prod07) and hybrid permeability (prod09). The range of permeability distributions is listed in Table 1. Each model generation consists of 3,000 simulations. Included in the dataset are the leakage rates determined from 2D wellbore models which utilize the pressure and CO2 saturation from LBL's reservoir simulations, NUFT mesh files with distributed lithology, NUFT rocktab files which describe the material properties for the geologic layers and the NUFT input files and post-processed output 'ntab' files. Each ntab file contains spatial (rows) and temporal (columns) model output tables for each model cell, the locations (x,y,z) and dimensions for each cells (dx, dy, dz). Table 1. Permeability distribution ranges for prod07 and prod09 model generations Geologic Layer: Permeability Range (log10 m^2) prod07 prod09 Etchegoin -12.92 to -10.92 -13.70 to -11.44 Macoma-Chanac -12.72 to -10.72 -13.50 to -11.24 Santa Margarita-McLure -12.70 to -10.70 -13.48 to -11.22 The input files used to generate the model include which are included in the dataset are: Time series of CO2 leakage input into the model (ex: Q_brn.W31-0.2.sim1000.layers123.tab) Time series of CO2 leakage input into the model (ex: Q_CO2.W31-0.2.sim1000.layers123.tab) Physical properties of the aquifer materials detailing the aquifer porosity, solid density, partitioning coefficients, permeabilities and van-Genuchten parameters detailed in a NUFT rocktab file: (ex: sim1000.usnt.rocktab) Numerical mesh and geologic data assigned to each model cell detailed in a NUFT genmsh format (ex: sim1000.mesh_k16.prod07.trans.genmsh) The primary output parameters are: pH (use absolute value) Change in TDS (mg/kg) Change in Pressure (Pa) Change CO2 gas saturation (fraction range 0.0-1.0) for example, the directory /p/lscratchh/mansoor1/nrap/kimberlina/prod09/mainfiles/sim1000/W31- 0.2 contains: sim1000.W31-0.2.trans.pH.red.ntab sim1000.W31-0.2.no_bg.trans.TDS.red.ntab sim1000.W31-0.2.usnt.P.deltabg.red.ntab sim1000.W31-0.2.usnt.CO2_sat.deltabg.red.ntab Each row in the NTAB files consist of model output per numerical grid cell. Each output file contains 33 columns (variables), including the information of numerical records, geologic location and sizes and the simulated parameter values over time. The first 13 variables are about numerical records and relative geologic information for a simulation grid: 1. index: simulation index 2. i: the ith grid of x-axis 3. j: the ith grid of y-axis 4. k: the ith grid of z-axis 5. element_ref: element reference 6. nuft_ind: nuft index 7. x: grid location in the x axis direction 8. y: grid location in the y axis direction 9. z: grid location in the z axis direction 10. dx: grid length in the x axis direction 11. dy: grid length in the y axis direction 12. dz: grid length in the z axis direction 13. volume: volume of the simulation grid The remainder (14, 15, 16...) variables are the simulated parameter values over time, take Pressure as an example, are: 14. 0.0y: initial pressure per cell. 15. 10.0y: simulated pressure at the end of the 10th year. 16. 20.0y: simulated pressure at the end of the 20th year. ... (repeated for every 10 years until 200 years)... The model extends 10,000 m, 5,000 m and 1,411 m in the x,y and z dimensions, respectively. The mesh consists of 164,832 cells with mesh dimensions of 101 x 51 x 32 (nx, ny, nz), with cell dimensions ranging from 100 m laterally (along x and y-axis) and model layers are as designated in the z-axis: Layer 1: atmosphere (1e-30 m thick) Layer 2: upper caprock (10 m thick) Layers 3-13: Etchegoin (536.23 m thck) Layers 14-27: Macoma-Chanac (679.04 m thick) Layers 28-32: Santa Margarita-McLure (185.94 m thick) The wellbore is placed along node i=51, j=26, and extends vertically along 5 nodes from the top to the bottom of the model. Special instructions when extracting files: Each Gzip archive (ex: prod07.sim1000-sim00099.tar.gz) contains 100 simulations. Gzip archives should be transferred into base directories (ie. In Linux: mkdir prod07; mv prod07.*.tar.gz prod07/.) before extracting, or files will be overwritten. Each sub-simulation tree should have the following file structure pattern (using the linux 'tree' command): |-- prod07 | |-- sim0001 | |-- W31-0.2 | | |-- Q_brn.W31-0.2.sim0001.layers123.tab | | |-- Q_co2.W31-0.2.sim0001.layers123.tab | | |-- sim0001.W31-0.2.no_bg.trans.TDS.red.ntab | | |-- sim0001.W31-0.2.trans.pH.red.ntab | | |-- sim0001.W31-0.2.usnt.CO2_sat.deltabg.red.ntab | | |-- sim0001.W31-0.2.usnt.P.deltabg.red.ntab | |-- W31-0.5 | | |-- Q_brn.W31-0.5.sim0001.layers123.tab | | |-- Q_co2.W31-0.5.sim0001.layers123.tab | | |-- sim0001.W31-0.5.no_bg.trans.TDS.red.ntab | | |-- sim0001.W31-0.5.trans.pH.red.ntab | | |-- sim0001.W31-0.5.usnt.CO2_sat.deltabg.red.ntab | | |-- sim0001.W31-0.5.usnt.P.deltabg.red.ntab | |-- W31-1.0 | | |-- Q_brn.W31-1.0.sim0001.layers123.tab | | |-- Q_co2.W31-1.0.sim0001.layers123.tab | | |-- sim0001.W31-1.0.no_bg.trans.TDS.red.ntab | | |-- sim0001.W31-1.0.trans.pH.red.ntab | | |-- sim0001.W31-1.0.usnt.CO2_sat.deltabg.red.ntab | | |-- sim0001.W31-1.0.usnt.P.deltabg.red.ntab | |-- sim0001.mesh_k16.prod07.trans.genmsh Disclaimer This document was prepared as an account of work sponsored by an agency of the United States government. Neither the United States government nor Lawrence Livermore National Security, LLC, nor any of their employees makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States government or Lawrence Livermore National Security, LLC. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States government or Lawrence Livermore National Security, LLC, and shall not be used for advertising or product endorsement purposes. Lawrence Livermore National Laboratory is operated by Lawrence Livermore National Security, LLC, for the U.S. Department of Energy, National Nuclear Security Administration under Contract DE-AC52-07NA27344. This report was reviewed and released as LLNL-MI-753464.

aquifer↗
DOE OSTI2020

Description of Sensor Assignment Optimization Method as Deployed on a Multi-Node Cluster

Data analytic methods are being developed to address the problem of how to assign a sensor set in a nuclear facility such that a requisite level of process monitoring capability is realized and that the sensor set is sufficiently rich to determine the status of the individual sensors with respect to need for calibration. There is an awareness in the nuclear industry that data analytics combined with rich sensor sets represent a means to improve operations and reduce costs. In the industry the calibration problem has been previously approached as an empirical data-driven problem with several methods having been developed. However, the experience of the utilities over the past ten years with these methods indicates that the absence of physics-based information renders the data-driven approach less reliable. Complicating factors such as the inherent variability of operation (both equipment alignment and operating condition) can confound a pure data-driven approach while there are no rigorous guidelines for determining what constitutes an adequate sensor set. The solution under development to overcome these shortcomings supplements the data analytic method with process information in a so-called process-constrained data-analytic approach. Simple balance equations are written for generic components (e.g., mechanical pump, valve, and heat exchanger). These do not require a priori knowledge of process parameters, such as heat transfer coefficients or friction factors. All that is needed on the part of the utility user is to identify the components and how they are connected. This report describes the development of a parallel computing capability for determining the optimal sensor set. The optimal sensor set problem suffers from the curse of dimensionality. Computation time increases exponentially as the size of the system grows. To overcome this difficulty a pre-conditioner algorithm is developed to find an approximate solution close the actual solution. This serves as a seed for the full-blown algorithm and acts to constrain the space that must searched. The optimization algorithms are described and the implementation on a parallel computing platform is described. The application of the method to a use case we are solving in collaboration with our utility partner served to illustrate how the default sensor set in a nuclear plant may not provide sufficient coverage to infer sensor calibration status.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗