Search NASA⌕ Search

SEARCH · Search NASA

Results for “Distributed File Systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey

Growing interest in Artificial Intelligence (AI) has resulted in a surge in demand for faster methods of Machine Learning (ML) model training and inference. This demand for speed has prompted the use of high performance computing (HPC) systems that excel in managing distributed workloads. Because data is the main fuel for AI applications, the performance of the storage and I/O subsystem of HPC systems is critical. In the past, HPC applications accessed large portions of data written by simulations or experiments or ingested data for visualizations or analysis tasks. ML workloads perform small reads spread across a large number of random files. This shift of I/O access patterns poses several challenges to modern parallel storage systems. In this paper, we survey I/O in ML applications on HPC systems, and target literature within a 6-year time window from 2019 to 2024. We define the scope of the survey, provide an overview of the common phases of ML, review available profilers and benchmarks, examine the I/O patterns encountered during offline data preparation, training, and inference, and explore I/O optimizations utilized in modern ML frameworks and proposed in recent literature. Lastly, we seek to expose research gaps that could spawn further R&D.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Demonstrating SolarPILOT’s Python API Through Heliostat Optimal Aimpoint Strategy Use Case

SolarPILOT is a software package that generates solar field layouts and characterizes the optical performance of concentrating solar power (CSP) tower systems. SolarPILOT was developed by the National Renewable Energy Laboratory (NREL) as a stand-alone desktop application but has also been incorporated into NREL’s1 System Advisor Model (SAM) in a simplified format. Prior means for user interaction with SolarPILOT have included the application’s graphical interface, the SAM routines with limited configurability, and through a built-in scripting language called “LK.” This paper presents a new, full-featured, Python-based application programmable interface (API) for SolarPILOT, which we hereafter refer to as CoPylot. CoPylot provides access to all SolarPILOT’s capabilities to generate and characterize power tower CSP systems seamlessly through Python. Supported capabilities include (i) creating and destroying a model instance with message reporting tools; (ii) accessing and setting any SolarPILOT variable including custom land boundaries for field layouts; (iii) programmatically managing receiver and heliostat objects with varied attributes for systems with multiple receiver or heliostat types; (iv) generating, assigning, and modifying solar field layouts including the ability to set individual heliostat locations, aimpoints, soiling rates, and reflectivity levels; (v) simulating solar field performance; (vi) returning detailed results describing performance of individual heliostats, the aggregate field, and receiver flux distribution; and, (vii) exporting Python-based model instances to multiple file formats. CoPylot enables Python users to perform detailed CSP tower analysis utilizing either the Hermite expansion technique (analytical) or the SolTrace ray-tracing engine. In addition to CoPylot’s functionality, Python users have access to the over 100,000 open-source libraries to develop, analyze, optimize, and visualize power tower CSP research. This enables CSP researchers to perform analysis that was previously not possible through SolarPILOT’s existing interfaces. This paper discusses the capabilities of CoPylot and presents a use case wherein we demonstrate optimal solar field aiming strategies.

41 EE - Solar Energy Technologies Office (EE-4S)↗
DOE OSTI2025

SCALE Input and Result Files Supporting SCALE Inventory and Reactivity Analysis of the gFHR

This dataset contains input and result files of computational simulations with the SCALE code system. The simulations cover radionuclide inventory and reactivity analyses of a fluoride salt-cooled high temperature pebble-bed reactor (PB-FHR), specifically the generic FHR benchmark. Users wanting to reproduce results from this dataset are required to obtain a license to the SCALE code system for which details on the distribution can be found here: https://www.ornl.gov/scale/releases

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2025

Continuing Development of the Nuclear Data Processing Code AMPX [Poster]

The ENDF/B-VIII.1 evaluation library has seen a great growth in the thermal neutron scattering sub-library. The SCALE code system has traditionally approached CE transport by assuming that the CE library on disk represented the fully expanded cumulative probability distributions, conditional on exiting angle and marginal on exiting energy. While this is a complete description of the data, it comes at the potential cost of large amounts of on-disk storage. This approach was strained by several TSL files in ENDF/B-VIII.1, such as graphite, which contained data for a large number of Bragg edges. In the fully expanded probability distributions, this was found to be a disproportionately large fraction of the SCALE CE library.

GNDS↗
DOE OSTI2020

UNITY: Unified Memory and Storage Space

UNITY is a 36-month project focused on providing design and evaluate a new distributed storage paradigm that unifies the traditionally distinct application views of memory- and file-based data storage into a single scalable and resilient environment. The project is a collaboration among Oak Ridge National Laboratory (ORNL, Lead institution), Los Alamos National Labs (LANL), and Georgia Tech (GT). The main contributions of the GT team have been around development of low level systems software for best leveraging the capabilities of new types of persistent memory technologies, and for development of methods for intelligent data management across memory/storage substrates with heterogeneous components. GT contributed the Phoenix library for optimized checkpoint/restart for HPC I/O for systems with non-volatile memory (NVM), the NVStream library for NVM-specialized streaming I/O for HPC workflows, the CoMerge, Mnemo and Kleio solutions for intelligent data management on NVM-based systems. These contributions result in significant improvements in both application performance and system efficiency.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Ducted Assembly Steady-State Heat Transfer Software (DASSH)

The Ducted Assembly Steady-State Heat transfer software (DASSH) is being developed at Argonne National Laboratory to perform a steady-state thermal fluids calculation to determine the coolant flow and temperature distribution for a hexagonal reactor core configuration with ducted assemblies. DASSH is intended to be used in the early stages of the reactor design process when assembly components can be undefined or are undergoing considerable design changes. DASSH provides a rapid assessment of the temperature and flow distribution to allow quick characterization of the system and identification of problem areas. This document is a guide for DASSH users. It summarizes the core capabilities of DASSH and highlights differences between DASSH and SE2-ANL. Instructions for obtaining and installing DASSH are provided along with some guidelines and recommendations for newer Python users. It covers the structure of the input file, provides directions for running DASSH, describes the various output files produced by the code, and provides instructions for visualizing data. Examples demonstrating various DASSH options are included. DASSH is under active development, so it is anticipated that this guide will grow and evolve as the code is updated.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2023

Energy Equity Opportunities in Distributed Wind Hybrid Systems for Rural Loads

Distributed wind (DW) hybrid systems, or combinations of distributed energy resources with wind technologies, have the potential to provide energy equity benefits for rural loads. However, project developers and end-users do not understand these benefits well. This report outlines energy equity opportunities that result directly from DW-hybrid projects and proposes a framework that demonstrates how to consider equity in the DW-hybrid system, which can enhance existing resilience and valuation tools by incorporating a way to include equity considerations. Energy equity opportunities achievable with DW-hybrid systems for rural loads are defined by the core principles of energy justice for use in a proposed equity framework. As well, a catalog of equity resources, metrics, technology-agnostic frameworks, and best practices are linked in an attached Excel file. Electric cooperatives, technical assistance providers (including national laboratory and DOE staff), DW project developers, and technology providers can use this work to understand, evaluate, and define equitable outcomes for potential DW-hybrid projects.

17 WIND ENERGY↗
DOE OSTI2025

Queued Up: 2025 Edition – Characteristics of Power Plants Seeking Transmission Interconnection As of the End of 2024 [Slides]

Electric transmission system operators (ISOs, RTOs, or utilities) require proposed power plants seeking to connect to the transmission grid to undergo a series of impact studies before they can be built. This process establishes what new transmission equipment or upgrades may be needed before a project can connect to the system and assigns the costs of that equipment. The lists of projects in this process are known as “interconnection queues”. In collaboration with interconnection.fyi, Berkeley Lab compiled, aggregated, and cleaned interconnection queue data from >50 transmission grid operators (7 ISO/RTOs and 49 non-ISO balancing areas), which collectively represent ~97% of currently installed U.S. electric generating capacity. The dataset includes requests submitted to queues through the end of 2024, and only includes requests seeking to connect to the transmission grid (not distribution-connected or behind-the-meter projects). The files below include both a PDF report and an Excel data file. The PDF report analyzes interconnection data and metrics through the end of 2024. The Excel data file includes (a) the full project-level interconnection queue dataset through 2024, (b) a codebook (data dictionary) describing each data field, and (c) 35 additional tabs featuring tables summarizing a range of interconnection metrics. Key highlights from the Queued Up: 2025 Edition (featuring data through 2024) include: • As of the end of 2024, there were ~10,300 projects actively seeking grid interconnection in the U.S., representing 1,400 GW of generation and approximately 890 GW of storage. • Historic withdrawal rates alongside relatively fewer new requests resulted in a 12% decrease in total active queue volume compared to the prior year. • Active natural gas capacity (136 GW, +72% year-over-year) increased in 2024, while solar (956 GW, -12%), storage (890 GW, -13%), and wind (271 GW, -26%) capacity decreased. • 408 GW of capacity already has a draft or executed interconnection agreement (IA) but has not yet reached commercial operations. • The time projects spend in queues before reaching COD is increasing. For the regions with available data, the median duration from IR to COD has doubled from <2 years for projects built in 2000-2007 to over 4 years for those built in 2018-2024. • Ultimately, most of this proposed capacity will not be built. Only 13% of capacity that submitted interconnection requests from 2000-2019 had reached commercial operations by the end of 2024; 77% of that capacity had been withdrawn and 10% was still active. • FERC Order 2023 and various other reforms are being implemented. These are important measures to reduce interconnection bottlenecks and enhance grid system reliability, but it is too early to measure and assess their full impact. • New additions for the 2025 edition include: (a) additional detail on data processing and gaps; (b) updates on interconnection reforms; (c) new analysis on interconnection agreements, and more.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

The git based ATLAS data acquisition configuration service in LHC Run 3

The ATLAS experiment at the LHC at CERN uses a large, distributed trigger and data acquisition system composed of many computing nodes, networks, and hardware modules. Its configuration service is used to provide descriptions of control, monitoring, diagnostic, recovery, dataflow and data quality configurations, interconnections, and parameters for modules, chips, and channels of various online systems, detectors, and the whole ATLAS experiment. Those descriptions have historically been stored in more than one thousand interconnected XML files, which are updated by various experts many times per day. Maintaining error-free and consistent sets of such files and providing reliable and fast access to current and historical configurations is a major challenge. This paper gives details of the configuration service upgrade on the modern Git version control system backend for LHC Run 3 and its exploitation experience. It may be interesting for developers using human-readable file formats, where consistency of the files, performance, access control, traceability of modifications, and effective archiving are key requirements.

Soloviev, Igor [Univ. of California, Irvine, CA (U↗
DOE OSTI2024

Continuous soil temperature measurements from 2019-10-4 to 2020-10-4, Teller road Mile 27, Seward Peninsula, Alaska

The dataset contains depth-resolved soil temperature measured at 45 discrete locations in a watershed located along the Nome-Teller road at Mile 27 in Seward Peninsula, Alaska. The dataset was generated to understand the local heterogeneity of soil thermal dynamics and their controls in a discontinuous permafrost region. At each location, temperatures were measured by a distributed temperature profiling probe designed based on Dafflon et al (2022). The dataset includes a description of the probe locations in the "Probe_locations.csv" file and the 45 data files (Soil_temperatures_*.csv). Metadata files include data descriptions (_dd.csv) for tabular data. All included files are listed and described in NGA513_flmd.csv.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Maps of plant functional type (PFT), PFT fractional cover, and uncertainty derived from AVIRIS-NG data, 2019, Seward Peninsula

Remote sensing maps of plant functional type (PFT), PFT fractional cover (fCover), and fCover uncertainty maps derived from NASA’s Airborne Visible / Infrared Imaging Spectrometer – Next Generation (AVIRIS-NG). The AVIRIS-NG imaging spectroscopy data (380 ~ 2510 nm) was collected as a part of the collaboration between NASA’s Arctic-Boreal Vulnerability Experiment (ABoVE; Miller et al., 2019) and DOE’s Next Generation Ecosystem Experiment in the Arctic (NGEE-Arctic). This package includes maps for flights over Teller, Kougarok, and Council watersheds created using eight AVIRIS-NG imagery collected in July, 2019. The map data and metadata are provided as image (ENVI, *.png) and text (*.txt, *hdr) formats. Additional supporting map quicklooks are provided as GIS *.kml files. Detailed description of the methods for each map are provided in Yang et al., (2023). These datasets are provided in support of Yang et al., (In revision) “Topography and Functional Traits Control the Distribution of Key Shrub Plant Functional Types in Low-Arctic Tundra”.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

ARDP Natrium Neutronic Methodology: Argonne Neutronic Assessment of ABR-1000

The Natrium Sodium-cooled Fast Reactor (SFR) concept developed by TerraPower, LLC was selected as one of the advanced reactor designs for demonstration under the Advanced Reactor Demonstration Program Natrium Demo project. In collaboration with TerraPower, the Argonne National Laboratory (ANL) team provide independent reviews of the neutronic methodology being used for the Natrium design. As part of this activity, both the TerraPower and ANL teams agreed to perform independent neutronics analyses of the ABR-1000 reactor problem as a demonstration on the usage of the neutronics methodologies. Comparison of the results from TerraPower and ANL would expose any methodological differences in the modeling approaches. This report describes and presents the Argonne analysis work on the ABR-1000, and will be provided to TerraPower, along with data files, to complete the comparison study. The comparison will cover the methodology to evaluate the reactivity coefficients for a safety analysis, the shutdown margin, and the reconstructed pin power distribution. The methods and models used to calculate the kinetic parameters and the reactivity coefficients (density, Doppler, axial/radial expansion, etc.) are described in detail with recommendations of specific method options and modelling techniques. The control system reactivity worth and the excess reactivity due to temperature defect are calculated for the shutdown margin analysis. The method to evaluate the excess reactivity is discussed in detail. The pin power reconstruction methodology used in SE2ANL, SE2RCT, and DASSH are discussed in this report. The pin power distribution evaluated by different methods are compared and discussed in the report.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2024

Updates to the Hanford Soil Inventory Model (SIM Version 2) for FY 2023

The purpose of this environmental calculation file (ECF) is to document the updates made to the Soil Inventory Model version 2.1 (SIM-v2.1) (ECF-HANFORD-21-0073, Updates to the Hanford Soil Inventory Model (SIM Version 2) for FY 2021) in FY 2023. These updates to SIM-v2.1 are referred to as the SIM-v2.2 version. The following updates have been made: • Include inventory estimate for a new analyte • hexavalent chromium (Cr(VI)), separate from total chromium inventory • Revise the inventory discharged to the 216-U-10 and 216-T-4 Pond systems based on partitioning of waste streams discharges over time and space among influent ditches and ponds • Enhance the preprocessing and postprocessing of the input and output files Note that the methodology for estimating Cr(VI) inventories is based on ECF-200W-23-0040, Recommendations for Updating Liquid Discharged Inventory and Transport Modeling Parameters for Cumulative Impacts Evaluation of Hexavalent Chromium in the 200 West Area while the revision of inventory discharged to the 216-U-10 and 216-T-4 Pond systems is based on ECF-HANFORD-19-0032, Distribution of Infiltration in the 216-U-10, 216-B-3 Pond, and 216-T-4 Pond Systems 1944-1997.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

SCALE Input and Result Files Supporting SCALE Inventory and Reactivity Analysis as Part of the Hermes 2021 PSAR Review

This dataset contains input and result files of computational simulations with the SCALE code system. The simulations cover radionuclide inventory and reactivity analyses of a fluoride salt-cooled high temperature pebble-bed reactor (PB-FHR), specifically the Hermes low-power PB-FHR demonstration reactor. Users wanting to reproduce results from this dataset are required to obtain a license to the SCALE code system for which details on the distribution can be found here: https://www.ornl.gov/scale/releases

equilibrium core↗
DOE OSTI2025

Disentangling the Impacts of Microtopography and Shrub Distribution on Snow Depth in a Subarctic Watershed: Toward a Predictive Understanding of Snow Spatial Variability: Supporting Data and Code

This repository contains R code and associated datasets for reproducing the analysis described in the manuscript titled “Disentangling the Impacts of Microtopography and Shrub Distribution on Snow Depth in a Subarctic Watershed: Toward a Predictive Understanding of Snow Spatial Variability” (DOI: 10.1029/2024JG008604). The provided scripts facilitate a comprehensive analysis of snow depth variability influenced by microtopography and vegetation distribution in a subarctic watershed. Included datasets are high-resolution spatial maps of snow depth, terrain elevation, vegetation height, and distance from shrubs taller than 1 meter, all formatted as text files (.txt). These data are fully describe in doi:10.15485/2316038. Users can adapt the provided R scripts to accommodate different data formats or larger spatial domains, noting that some output files may require modification due to their size.The code includes implementations for boosted regression tree analysis adapted from methods outlined in Elith et al. (2008). Users interested in understanding or modeling landscape-scale snow distribution patterns, particularly in Arctic or subarctic ecosystems, will find this package useful. The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Distribution of Infiltration in the 216-U-10, 216-B-3 Pond, and 216-T-4 Pond Systems 1944-1997

A modeling system to simulate fate and transport of contaminant plumes in groundwater beneath the Central Plateau to support the Cumulative Impact Evaluation (CIE) and Composite Analysis (CA) is being developed. The CIE will evaluate the cumulative effect of multiple Central Plateau radiological and chemical sources on groundwater quality. This will enable cleanup decisions to be evaluated in the context of all Central Plateau waste sites and existing groundwater conditions. The purpose of this Environmental Calculation File (ECF) is to partition the infiltration of wastewater and contaminant inventory between the main pond and each expansion lobe of the 216-B-3 Pond system, and to partition the infiltration of wastewater and contaminant inventory between the ditches and ponds of the 216-U-10 Pond and 216-T-4 Pond systems.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2025

BULKI-Store v0.3.2

BULKI-Store is a distributed object storage system optimized for high-performance computing environments. Built with a Rust core and Python bindings, it efficiently manages scientific and machine learning datasets across HPC clusters. The system employs a client-server architecture with MPI integration, enabling seamless scaling on supercomputers like Perlmutter. BULKI-Store's object-oriented approach provides intuitive data organization with rich metadata support, contrasting with traditional file-based solutions. Key optimizations include selective checkpoint loading, unified checkpoint files, and object chunking for large data transfers. For machine learning workloads, BULKI-Store offers advantages through fine-grained access patterns, dynamic data sharing between training instances, and reduced memory pressure. Memory management features include strategic Python GC calls, minimized data copies, and batch processing capabilities. The system leverages Rayon's thread pool for asynchronous data prefetching and supports multiple CPU architectures (ARM64, x86, AMD, RISC-V). By combining performance optimizations with developer-friendly APIs, BULKI-Store addresses the complex data management challenges of modern HPC applications while maintaining compatibility across heterogeneous computing environments.

Zhang, Wei [Lawrence Berkeley National Laboratory ↗
DOE OSTI2026

Fracture Intersections under Stress: Laboratory Data and Code [Data set]

The connectivity of natural and induced fractures governs the injection and withdrawal of fluids from subsurface reservoirs. Connectivity depends on intersections that control how fluids mix and move through the entire system. Here, we present data sets from 3D X-ray microscopy measurements of simple fracture networks under stress. 3D printing was used to create prismatic blocks that formed fracture networks composed of 2 orthogonal fractures. The network orientation was either "x" or "+" relative to an applied vertical stress. 3D data sets were collected for normal loads of 25, 100 and 200 Newtons for samples with fracture surfaces with either correlated or uncorrelated asperity distributions. The file contains data from the 12 samples analyzed along with an example code used to extract the intersection geometry. Additional experimental details can be found in the manuscript "Geologic Stress Modulates Fluid Mixing at Fracture Intersections" (10.1038/s43247-026-03525-9)and supplemental information to appear in Communications Earth & Environment in 2026.

02 PETROLEUM↗