Search NASA⌕ Search

SEARCH · Search NASA

Results for “File System”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 865 records · Page 48

CO2 Enhanced Oil Recovery Evaluation System (CO2_E_EvSystem), Version 2025

The United States Department of Energy’s (DOE’s) Office of Fossil Energy and the National Energy Technology Laboratory (NETL) developed a suite of screening-level, techno-economic models/tools, known as the CO2_E_EvSystem, to evaluate technical aspects and costs of using carbon dioxide (CO2) enhanced oil recovery (EOR) to store CO2 and produce oil. CO2_E_EvSystem has three software components and several input and output files. The software components are CO2_E_EvTool, CO2_Prophet, and CO2_E_COM. Almost all computational work is performed by CO2_Prophet and CO2_E_COM, which are both Fortran programs. The primary role of CO2_E_EvTool is to manage input and output files for the two programs, run the two programs, allow multiple oilfields to be evaluated in a single run, and generate files that summarize the results for all the oilfields run. This version of the CO2_E_EvSystem includes a residual oil zone dataset from the San Andres formation, Permian Basin, to demonstrate the system and guide users on attributes needed for dataset inputs.

AS↗

NCSP FY 2015 AMPX Annual Report

The objective of the Nuclear Criticality Safety Program (NCSP) Analytical Methods program element is to sustain state-of-the-art radiation transport modeling capabilities and the expertise necessary to develop, maintain, and disseminate the analytical tools and data libraries in a manner that is responsive to the needs of those responsible for developing, implementing, and maintaining criticality safety. To this end, the objective of the AMPX Development and Maintenance Subtask is to ensure that the AMPX cross-section processing system is available to prepare cross-section and covariance libraries that are needed by SCALE and other radiation transport packages to support criticality safety analyses of systems with fissionable material. Moreover, the task ensures that the cross-section processing software is current and up-to-date with the Evaluated Nuclear Data File (ENDF/B) formats and procedures for disseminating evaluated cross-section data provided through new measurement and evaluation work efforts. The AMPX subtask provides for the production of data libraries needed for criticality safety analyses. As such, the task provides the “bridge” between the nuclear data evaluations and the radiation transport software. In the absence of the AMPX cross-section processing task, the ability to maintain and update the SCALE nuclear data libraries (cross-section and covariance data) would be hindered. Therefore, the AMPX Development and Maintenance Subtask ensures that SCALE is up-to-date with the latest cross-section data for supporting criticality safety analyses. This letter report provides a summary of the FY 2015 AMPX development and maintenance activities that have been performed in support of the NCSP. Specifically, the report summarizes the AMPX code system updates and modernization accomplishments that have been made to the software package. Also, the report documents the cross-section library updates that have been made for the SCALE radiation transport package. In addition, the report documents testing activities that have been performed to process available cross-section and covariance data files to ensure the latest ENDF/B data files can be processed properly. This report satisfies the following reporting milestone in the NCSP FY16-20 Five Year Plan: Document AMPX modernization and technical support for SCALE continuous energy (CE), multigroup (MG), and covariance libraries and report status annually to the NCSP Manager (AM3: Q2)

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Geochemistry, volumetric water content, and active layer depths from rainfall simulations at the Kougarok Fire Site, Seward Peninsula, Alaska, 2022

Rainfall frequency and intensity is expected to increase in the Arctic, with potential implications for permafrost thaw and geochemical fluxes in soils. To conduct controlled rainfall experiments at remote field sites in the Arctic, the Next-Generation Ecosystem Experiments Arctic Rainfall Simulator (NARS) was developed at Los Alamos National Laboratory as part of the NGEE Arctic project. To better understand how rainfall may affect interflow biogeochemistry and permafrost thaw, rainfall simulations were performed on the Seward Peninsula of Alaska during late September and early October 2022 at the Kougarok Fire Site near mile marker 86 of the Nome-Taylor Highway. Water samples were collected before, during, and after rainfall simulations until interflow had ceased. Additional data collected included soil pore water samples from macrorhizons, active layer depths, and volumetric water content at each of the experimental plots. This dataset contains one *.csv file of water sample properties, one *.csv file of thaw depths, one *.csv file of soil moisture measurements, and one *.kml file of the locations where samples were collected.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy’s Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy’s Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗

Leaf area index (LAI), Teller site, Seward Peninsula, Alaska, 2022

Leaf area index (LAI) sampled from locations on the Teller MM27 NGEE-Arctic site, Seward Peninsula, Alaska. These data were collected in support of the ongoing NGEE-Arctic and NASA ABoVE data synthesis work. Samples were collected in July 2022 at 100 locations that cover the down slope half of Teller MM27 hillslope. This data package includes sample information and LAI data (.csv), an ESRI shape file (.shp) and Keyhole Markup Language (.kml) that define the location and area of each LAI measurement. Metadata files include data descriptions (_dd.csv) for tabular data, file level metadata (.csv), and methods and the instrument manual (.pdf). The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗

NEPATEC2.0: NEPA Text Corpus v2.0

The National Environmental Policy Act of 1969, as amended (NEPA), is a major environmental law in the United States, requiring Federal agencies to consider and document potential environmental impacts before deciding on a proposed action. Modernization of NEPA and permitting processes faces significant challenges due to the lack of standardized formats and interoperable systems for organizing and sharing NEPA-related information across agencies. Much of the information gathered during NEPA reviews is written into documents such as categorical exclusions, environmental assessments, and environmental impact statements, then filed in predominately independent agency file stores that may or may not be publicly accessible. The application of metadata and data standards, such as those recommended by the Council on Environmental Quality (CEQ), to NEPA documents offers a shared vocabulary and structure for key entities like projects, processes, and documents that can streamline information exchange and enhance collaboration across systems. In this work, we publicly release NEPATEC2.0, an expanded corpus of NEPA documents with associated metadata. NEPATEC2.0 encompasses approximately 120,000 documents from 60,000 projects prepared by more than 60 different agencies. Modeled to align with CEQ metadata standards, NEPATEC2.0 promotes consistency in environmental reviews and supports the ongoing effort to modernize permitting technologies by facilitating more transparent, efficient, and data-driven decision-making. Importantly, NEPATEC2.0 demonstrates the possibilities and limitations of large language model-based prompting to extract information from NEPA documents at scale.

environmental review↗

NEPATEC v2.0: Standardized Metadata and Text Corpus of National Environmental Policy Act Documents

The National Environmental Policy Act of 1969, as amended (NEPA), is a major environmental law in the United States, requiring Federal agencies to consider and document potential environmental impacts before deciding on a proposed action. Modernization of NEPA and permitting processes faces significant challenges due to the lack of standardized formats and interoperable systems for organizing and sharing NEPA-related information across agencies. Much of the information gathered during NEPA reviews is written into documents such as categorical exclusions, environmental assessments, and environmental impact statements, then filed in predominately independent agency file stores that may or may not be publicly accessible. The application of metadata and data standards, such as those recommended by the Council on Environmental Quality (CEQ), to NEPA documents offers a shared vocabulary and structure for key entities like projects, processes, and documents that can streamline information exchange and enhance collaboration across systems. In this work, we publicly release NEPATEC2.0, an expanded corpus of NEPA documents with associated metadata. NEPATEC2.0 encompasses approximately 120,000 documents from 60,000 projects prepared by more than 60 different agencies. Modeled to align with CEQ metadata standards, NEPATEC2.0 promotes consistency in environmental reviews and supports the ongoing effort to modernize permitting technologies by facilitating more transparent, efficient, and data-driven decision-making. Importantly, NEPATEC2.0 demonstrates the possibilities and limitations of large language model-based prompting to extract information from NEPA documents at scale.

54 ENVIRONMENTAL SCIENCES↗

Resource Management for Processing Wide Area Data Streams on Supercomputers

Modern scientific instruments generate enormous amount of data. Typically, the data collected from the instruments are stored in one or more files that are then moved to a distant supercomputer for processing. The final results are sent back to the user. In order to make effective use of the time on expensive instruments, experimenters want to process the data as they are generated. They want to stream the data from instruments' memory directly to a supercomputer's memory for analysis. Since the compute nodes in a supercomputer are not connected directly to the wide area network, the data streams need to be passed through intermediate gateway nodes. As opposed to the best effort file transfers, data streaming applications require resources at a specific time for a specific period. In this paper, we present a system model for enabling data streaming through gateway nodes and an algorithm to efficiently allocate gateway node resources along with compute nodes. We evaluate the algorithm using real-world traces on the Chameleon Cloud. The results show that our system can schedule compute and gateway resources efficiently for streaming analysis.

data streaming↗

SU-F-I-11: Software Development for 4D-CBCT Research of Real-Time-Image Gated Spot Scanning Proton Therapy

Purpose: To acquire correct information for inside the body in patient positioning of Real-time-image Gated spot scanning Proton Therapy (RGPT), utilization of tomographic image at exhale phase of patient respiration obtained from 4-dimensional Cone beam CT (4D-CBCT) has been desired. We developed software named “Image Analysis Platform” for 4D-CBCT researches which has technique to segment projection-images based on 3D marker position in the body. The 3D marker position can be obtained by using two axes CBCT system at Hokkaido University Hospital Proton Therapy Center. Performance verification of the software was implemented. Methods: The software calculates 3D marker position retrospectively by using matching positions on pair projection-images obtained by two axes fluoroscopy mode of CBCT system. Log data of 3D marker tracking are outputted after the tracking. By linking the Log data and gantry-angle file of projection-image, all projection-images are equally segmented to spatial five-phases according to marker 3D position of SI direction and saved to specified phase folder. Segmented projection-images are used for CBCT reconstruction of each phase. As performance verification of the software, test of segmented projection-images was implemented for sample CT phantom (Catphan) image acquired by two axes fluoroscopy mode of CBCT. Dummy marker was added on the images. Motion of the marker was modeled to move in 3D space. Motion type of marker is sin4 wave function has amplitude 10.0 mm/5.0 mm/0 mm, cycle 4 s/4 s/0 s for SI/AP/RL direction. Results: The marker was tracked within 0.58 mm accuracy in 3D for all images, and it was confirmed that all projection-images were segmented and saved to each phase folder correctly. Conclusion: We developed software for 4D-CBCT research which can segment projection-image based on 3D marker position. It will be helpful to create high quality of 4D-CBCT reconstruction image for RGPT.

60 APPLIED LIFE SCIENCES↗

A Python Module for Storing CCD Images with openPMD (openPMD-CCD) v0.1.0

openPMD is an open meta-data schema that provides meaning and self-description to data sets in science and engineering. The openPMD-CCD software module adds interfaces to organize camera (CCD) images in hierarchical data files. This software provides modern I/O storage formats from high-performance computing and provides bindings for integration into experimental control systems, e.g. via Python 3 and LabView 2020.

Gonsalves, AnthonyJ↗

mrizwanriaz/lcm-dataset

Laser capture microdissection (LCM) derived stem cell-type transcriptome (Jie et al.) How to run on your system Install R on your system. Install the required packages shiny, shinyjs, ggplot2, dplyr, formattable, rcartocolor Download the RShiny app code (ui.R and server.R) and RDS files from the GitHub repository to the same directory. Usage: Run the R code file (ui.R) in the RStudio. The app will open in the default web browser.

Riaz, Muhammad Rizwan↗

Web Extensible Display Manager 2

The Web Extensible Display Manager (WEDM) was first deployed at Jefferson Lab (JLab) in 2016 with the goal of rendering Extensible Display Manager (EDM) control screens on the web for the benefit of accessibility, and with version 2 our aim is to provide a more general purpose display toolkit by freeing ourselves from the constraints of the EDM dependency. Over the last few years WEDM has been extensively used at JLab for 24/7 information kiosks, on-call monitoring, and by remote users and staff. The software has also been deployed to Oak Ridge National Laboratory, and has become more robust as many bug fixes and contributions have been added. However, adoption and utility of the software as a general purpose control system display manager is limited by EDM, which is no longer actively maintained. A new toolkit can be built on modern frameworks, fully embrace web conventions and standards, and support multiple control system data sources. This new version is a result of a technology review and selection, and introduces a web inspired display file format, a web based display builder, new widgets, and a data interface intended to support pluggable data.

Slominski, R. J.↗

Utility-Scale Solar, 2021 Edition: Empirical Trends in Deployment, Technology, Cost, Performance, PPA Pricing, and Value in the United States [Slides]

Berkeley Lab’s “Utility-Scale Solar, 2021 Edition” provides an overview of key trends in the U.S. market, with a focus on 2020. Highlights of this year’s update include: A record of nearly 9.6 GWAC of new utility-scale PV capacity came online in 2020, bringing cumulative installed capacity to more than 38.7 GWAC across 43 states. 89% of all new utility-scale PV capacity added in 2020 uses single-axis tracking. Median installed project costs declined to $\$$1.4/WAC (or $\$$1.1/WDC) in 2020. Project-level capacity factors vary widely, from 9% to 36% (on an AC basis), with a sample median of 24%. The report explores drivers of this variation. Utility-scale PV’s LCOE fell to $\$$34/MWh in 2020 ($\$$28/MWh if factoring in the federal investment tax credit, or ITC). PPA prices have largely followed the decline in solar’s LCOE over time, but have stagnated more recently. Prices from a sample of recent contracts average just above $\$$20/MWh (levelized). In 2020, solar’s average market value (defined in the report to include only energy and capacity value) exceeded average wholesale prices in 12 of the 17 balancing authorities analyzed (including 4 of the 7 independent system operators across the United States). Adding battery storage is one way to increase the value of solar. Our public data file tracks metadata for more than 150 PV+battery hybrid projects that are already online or that have secured offtake arrangements. At the end of 2020, there were at least 460 GW of utility-scale solar power capacity within the interconnection queues across the nation, 160 GW of which include batteries. For more information, and to explore related interactive data visualizations, go to utilityscalesolar.lbl.gov.

14 SOLAR ENERGY↗

Zero Power Warming (ZPW) Chamber Prototype Measurements, Utqiagvik (Barrow), Alaska, 2016

Data were collected during one season of prototyping associated with the development of a passive warming technology. An experimental chamber, the Zero Power Warming (ZPW) chamber, was fitted with apparatus to modulate venting of a field enclosure and enhance elevation of air temperature by solar radiation. The ZPW chamber was compared with a control chamber (Control) and an ambient open air plot (Ambient). The control chamber was identical to the ZPW chamber but lacked the apparatus necessary to modulate venting, the chamber vents in the control chamber were fixed open for the majority of the trial period. The three plots were located over Carex aquatilis growing in an area of moderately degraded permafrost. Chambers were placed on the same footprints that were used for a similar exercise in 2015 (no data) and therefore those plots had experienced some thaw and degradation prior to 2016. The following data were collected for 80 days at 1 minute intervals from within two chambers and an ambient plot: solar input, chamber venting, air temperature, relative humidity, soil temperature (at 5, 10 and 15 cm), soil moisture, downward and upward NIR. CO2 concentration within the chambers and of the ambient air plot was collected at 1 minute intervals over a 62 hour period in the middle of the growing season. This dataset contains six csv files and one pdf file.The documentation may refer to a ZPW-metadata.xlsx file. This file was split into two csv files: ZPW-metadata.csv and ZPW-metadata-instrumentation.csv.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗

NGEE Arctic Plant Traits: Soil Temperature and Moisture, Kougarok Road Mile Marker 64, Seward Peninsula, Alaska, 2016-2022

Data includes soil temperature and soil moisture measured at the Kougarok hill slope located at Kougarok Road, Mile Marker 64. Most measurements are from monitoring stations with permanently installed probes though the data also includes single point measurements from handheld devices. Data collection began in July 2016 and was completed October 2022. The final version of this dataset was archive in 2023. This dataset includes two csv files and one pdf.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗

Modeling Anaerobic Soil Organic Carbon Decomposition in Arctic Polygon Tundra: Insights into Soil Geochemical Influences on Carbon Mineralization: Modeling Archive

This anaerobic carbon decomposition model framework was developed with explicit representation of fermentation, methanogenesis and iron reduction by combining three well-known modeling approaches developed in different disciplines. A pool-based model to represent upstream carbon transformations and replenishment of dissolved organic carbon (DOC) pool, a thermodynamically-based model to calculate rate kinetics and biomass growth for methanogenesis and Fe(III) reduction, and a humic ion-binding model for aqueous phase speciation and pH calculations. The combined models are implemented in the open source geochemical model PHREEQC (V3.0). Installation of PHREEQC is required to run this model. This dataset contains one zipped file and one pdf.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗

Mercury Methylation and Soil Organic Matter Degradation in Incubations with Added Acetate, Isotope-Labeled Mercury, and Selective Microbial Inhibitors, Teller and Council Sites, Seward Peninsula, 2018-2019

This dataset provides the results of laboratory soil microcosm incubation experiments conducted under anaerobic conditions with added acetate, isotope-labeled mercury, and selective microbial inhibitors (2-bromoethanesulfonate and sodium molybdate). The microcosms contained soil collected in August 2018 from the NGEE Arctic Teller Road Site at mile marker 27 (TL_MM27) near Intensive Site 9 (TL_IS_9) and from the Council Road Site (CN_MM71). Incubations were conducted at 8°C for 180 days (Teller incubations) or 60 days (Council incubations). The microcosm headspace was sampled at designated intervals for greenhouse gas concentrations, then destructively harvested and extracted using ultrapure water and 0.1 M KCl for analysis of the porewater geochemistry. This dataset contains one pdf, two csv data files, and one zipped file with raw data.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗

Accelerating Scientific Workflows on HPC Platforms with In Situ Processing

Scientific workflows drive most modern large-scale science breakthroughs by allowing scientists to define their computations as a set of jobs executed in a given order based on their data dependencies. Workflow management systems (WMSs) have become key to automating scientific workflows-executing computational jobs and orchestrating data transfers between those jobs running on complex high-performance computing (HPC) platforms. Traditionally, WMSs use files to communicate between jobs: a job writes out files that are read by other jobs. However, HPC machines face a growing gap between their storage and compute capabilities. To address that concern, the scientific community has adopted a new approach called in situ, which bypasses costly parallel filesystem I/O operations with faster in-memory or in-network communications. When using in situ approaches, communication and computations can be interleaved. In this work, we leverage the Decaf in situ dataflow framework to accelerate task-based scientific workflows managed by the Pegasus WMS, by replacing file communications with faster MPI messaging. We propose a new execution engine that uses Decaf to manage communications within a sub-workflow (i.e., set of jobs) to optimize inter-job communications. We consider two workflows in this study: (i) a synthetic workflow that benchmarks and compares file- and MPI-based communication; and (ii) a realistic bioinformatics workflow that computes mu-tational overlaps in the human genome. Experiments show that in situ communication can improve the bioinformatics workflow execution time by 22% to 30% compared with file communication. Our results motivate further opportunities and challenges for bridging traditional WMSs with in situ frameworks.

Decaf↗

BULKI-Store v0.3.2

BULKI-Store is a distributed object storage system optimized for high-performance computing environments. Built with a Rust core and Python bindings, it efficiently manages scientific and machine learning datasets across HPC clusters. The system employs a client-server architecture with MPI integration, enabling seamless scaling on supercomputers like Perlmutter. BULKI-Store's object-oriented approach provides intuitive data organization with rich metadata support, contrasting with traditional file-based solutions. Key optimizations include selective checkpoint loading, unified checkpoint files, and object chunking for large data transfers. For machine learning workloads, BULKI-Store offers advantages through fine-grained access patterns, dynamic data sharing between training instances, and reduced memory pressure. Memory management features include strategic Python GC calls, minimized data copies, and batch processing capabilities. The system leverages Rayon's thread pool for asynchronous data prefetching and supports multiple CPU architectures (ARM64, x86, AMD, RISC-V). By combining performance optimizations with developer-friendly APIs, BULKI-Store addresses the complex data management challenges of modern HPC applications while maintaining compatibility across heterogeneous computing environments.

Zhang, Wei [Lawrence Berkeley National Laboratory ↗