Search NASA⌕ Search

SEARCH · Search NASA

Results for “HDF5”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

LowFive v1.0

LowFive is a new data transport layer based on the HDF5 data model, for in situ workflows. Executables using LowFive can communicate in situ (using in-memory data and MPI message passing), reading and writing traditional HDF5 files to physical storage, and combining the two modes. Minimal and often no source-code modification is needed for programs that already use HDF5. LowFive maintains deep copies or shallow references of datasets, configurable by the user. More than one task can produce (write) data, and more than one task can consume (read) data, accommodating fan-in and fan-out in the workflow task graph. LowFive supports data redistribution from n producer processes to m consumer processes.

Morozov, Dmitriy↗
DOE OSTI2017

Seven years (2008-2014) of meteorological observations plus a synthetic El Nino drought for BCI Panama.

This dataset is a derivative product of raw meteorological data collected at Barro Colorado Island, Panama (see acknowledgements below). This dataset contains the following: 1) a seven-year record (2008-2014) of meteorological observations from BCI that is in a comma delimited text format, 2) an R-script that converts the observed meteorology into an hdf5 format that can be read by the ED2 model, 3) two decades of meteorological drivers in hdf5 format that are based on the 7-year record of observations and include a synthetic 2-yr El Nino drought, 4) a ReadMe.txt file that explains how the data in the hdf5 meteorological drivers correspond to the observations. The raw meteorological data were further QC'd as part of the NGEE-Tropics project to derive item 1 above. The R-script makes the appropriate unit conversions for all observed meteorological variables to be compatible with the ED2 model. The R-script also converts RH into specific humidity, splits total shortwave radiation into its 4-stream parts, and calculates longwave radiation from air temperature and RH. The synthetic El Nino drought is based on selected months from the observed meteorology where in each, precipitation (only) of the selected months was modified to reflect the precipitation patterns of the 1982/83 El Nino observed at BCI.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

MicroBooNE BNB Electron Neutrino Overlay Sample (No Wire Info)

MicroBooNE samples are provided for collaborative development in two different formats: HDF5, targeting the broadest audience, and artroot, targeting users that are familiar with the software infrastructure of Fermilab neutrino experiments and more in general of HEP experiments. The HDF5 files are stored on Zenodo, together with a list of artroot files accessible with xrootd.This sample includes simulated interactions of neutrinos from the Booster Neutrino Beam (BNB), overlaid on top of cosmic ray data. The sample is restricted to charged-current electron neutrino interactions within the argon active volume of the time projection chamber.The HDF5 files in this sample do not include the information at the wire waveform level ("NoWire" label), allowing for larger number of events to be included in the data set.More documentation, including detailed description of content, recipes, and example usage, at https://github.com/uboone/OpenSamples.Suggested text for acknowledgment is the following:We acknowledge the MicroBooNE Collaboration for making publicly available the data sets [data set DOIs] employed in this work. These data sets consist of simulated neutrino interactions from the Booster Neutrino Beamline overlaid on top of cosmic data collected with the MicroBooNE detector [2017 JINST 12 P02017].In addition, we request that software products resulting from the usage of the datasets are also made publicly available.

Abratenko, Polina↗
DOE OSTI2022

MicroBooNE BNB Inclusive Overlay Sample (With Wire Info)

MicroBooNE samples are provided for collaborative development in two different formats: HDF5, targeting the broadest audience, and artroot, targeting users that are familiar with the software infrastructure of Fermilab neutrino experiments and more in general of HEP experiments. The HDF5 files are stored on Zenodo, together with a list of artroot files accessible with xrootd.This sample includes simulated interactions of neutrinos from the Booster Neutrino Beam (BNB), overlaid on top of cosmic ray data. The sample is inclusive, i.e. it includes all types of neutrinos and interactions, with relative abundance matching our nominal flux and cross section models. Interactions are simulated in in the whole cryostat volume.The HDF5 files in this sample include the information at the wire waveform level (after deconvolution and finding of regions of interest). As this information significantly increases the file size, this sample contains about 17% of the events of the corresponding sample without wire information.More documentation, including detailed description of content, recipes, and example usage, at https://github.com/uboone/OpenSamples.Suggested text for acknowledgment is the following:We acknowledge the MicroBooNE Collaboration for making publicly available the data sets [data set DOIs] employed in this work. These data sets consist of simulated neutrino interactions from the Booster Neutrino Beamline overlaid on top of cosmic data collected with the MicroBooNE detector [2017 JINST 12 P02017].In addition, we request that software products resulting from the usage of the datasets are also made publicly available.

Abratenko, Polina↗
DOE OSTI2022

MicroBooNE BNB Electron Neutrino Overlay Sample (With Wire Info)

MicroBooNE samples are provided for collaborative development in two different formats: HDF5, targeting the broadest audience, and artroot, targeting users that are familiar with the software infrastructure of Fermilab neutrino experiments and more in general of HEP experiments. The HDF5 files are stored on Zenodo, together with a list of artroot files accessible with xrootd.This sample includes simulated interactions of neutrinos from the Booster Neutrino Beam (BNB), overlaid on top of cosmic ray data. The sample is restricted to charged-current electron neutrino interactions within the argon active volume of the time projection chamber.The HDF5 files in this sample include the information at the wire waveform level (after deconvolution and finding of regions of interest). As this information significantly increases the file size, this sample contains about 20% of the events of the corresponding sample without wire information.More documentation, including detailed description of content, recipes, and example usage, at https://github.com/uboone/OpenSamples.Suggested text for acknowledgment is the following:We acknowledge the MicroBooNE Collaboration for making publicly available the data sets [data set DOIs] employed in this work. These data sets consist of simulated neutrino interactions from the Booster Neutrino Beamline overlaid on top of cosmic data collected with the MicroBooNE detector [2017 JINST 12 P02017].In addition, we request that software products resulting from the usage of the datasets are also made publicly available.

Abratenko, Polina↗
DOE OSTI2022

MicroBooNE BNB Inclusive Overlay Sample (No Wire Info)

MicroBooNE samples are provided for collaborative development in two different formats: HDF5, targeting the broadest audience, and artroot, targeting users that are familiar with the software infrastructure of Fermilab neutrino experiments and more in general of HEP experiments. The HDF5 files are stored on Zenodo, together with a list of artroot files accessible with xrootd.This sample includes simulated interactions of neutrinos from the Booster Neutrino Beam (BNB), overlaid on top of cosmic ray data. The sample is inclusive, i.e. it includes all types of neutrinos and interactions, with relative abundance matching our nominal flux and cross section models. Interactions are simulated in in the whole cryostat volume.The HDF5 files in this sample do not include the information at the wire waveform level ("NoWire" label), allowing for larger number of events to be included in the data set.More documentation, including detailed description of content, recipes, and example usage, at https://github.com/uboone/OpenSamples.Suggested text for acknowledgment is the following:We acknowledge the MicroBooNE Collaboration for making publicly available the data sets [data set DOIs] employed in this work. These data sets consist of simulated neutrino interactions from the Booster Neutrino Beamline overlaid on top of cosmic data collected with the MicroBooNE detector [2017 JINST 12 P02017].In addition, we request that software products resulting from the usage of the datasets are also made publicly available.

Abratenko, Polina↗
NASA NTRS2005

The Hierarchical Data Format for EOS (HDF-EOS)

HDF is a file format and a software library for data storage, management, exchange, and archiving. It is written and maintained by the National Center for Supercomputing Applications (NCSA). HDF5 has a very simple but versatile data model which is compatible with most competing formats. Through its grouping and linking mechanisms, the HDF5 data model enables complex data relationships and dependencies. HDF5 accommodates the inclusion of many common types of metadata and arbitrary types and quantities of user-defined metadata.

Ullman, Richard↗
DOE OSTI2017

ArrayBridge: Interweaving declarative array processing with high-performance computing

Scientists are increasingly turning to datacenter-scale computers to produce and analyze massive arrays. Despite decades of database research that extols the virtues of declarative query processing, scientists still write, debug and parallelize imperative HPC kernels even for the most mundane queries. This impedance mismatch has been partly attributed to the cumbersome data loading process; in response, the database community has proposed in situ mechanisms to access data in scientific file formats. Scientists, however, desire more than a passive access method that reads arrays from files. This paper describes ArrayBridge, a bi-directional array view mechanism for scientific file formats, that aims to make declarative array manipulations interoperable with imperative file-centric analyses. Our prototype implementation of ArrayBridge uses HDF5 as the underlying array storage library and seamlessly integrates into the SciDB open-source array database system. In addition to fast querying over external array objects, ArrayBridge produces arrays in the HDF5 file format just as easily as it can read from it. ArrayBridge also supports time travel queries from imperative kernels through the unmodified HDF5 API, and automatically deduplicates between array versions for space efficiency. Our extensive performance evaluation in NERSC, a large-scale scientific computing facility, shows that ArrayBridge exhibits statistically indistinguishable performance and I/O scalability to the native SciDB storage engine.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Web-based Preprocessing and Visualization of 3D FIB Tomography Data for Nuclear Fuel Characterization

Three-dimensional (3D) focused ion beam (FIB) tomography enables reconstruction of internal nuclear fuel features that can't be fully evaluated through surface imaging alone. This capability supports characterization of fuel constituents and defects under thermal and irradiation conditions relevant to microreactor development. However, large tomography datasets can create data-handling, loading, and visualization challenges, especially when image-stack preparation and file conversion must be completed with separate tools. The Computational Ultraspatial Tomography Toolkit for High-Resolution Object Analysis Tools (CUTTRHOAT) is an open-source web application being developed to display FIB tomography datasets available through the Nuclear Research Data System (NRDS). The current alpha version requires prepared HDF5 datasets and has limited integrated data-preparation capabilities. This project improves CUTTHROAT by adding dataset-folder selection, automatic input detection, dataset scanning, missing-slice identification, blank-slice insertion, and image-stack-to-HDF5 conversion. Two applications will be compared: the baseline CUTTHROAT alpha workflow and the updated application containing the integrated data-handling and preprocessing functions. Evaluation will consider dataset detection accuracy, conversion success, loading time, rendering responsiveness, application stability, and user interaction. Preliminary results demonstrate successful loading of existing HDF5 files and converted image stacks, while testing also identified performance reductions caused by excessive blank-slice generation. The updated workflow reduces reliance on external preparation tools and supports more direct movement from image stacks to color-code 3D visualization. Future work includes refining missing-slice handling, integrating additional preprocessing functions, like a denoising feature, parsing TIFF metadata for automatic voxel scaling, and adding manual X, Y, and Z voxel-spacing inputs for PNG and JPEG.

36 - MATERIALS SCIENCE↗
DOE OSTI2020

ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems

Scientific applications at exascale generate and analyze massive amounts of data. A critical requirement of these applications is the capability to access and manage this data efficiently on exascale systems. As such, parallel I/O, the key technology enables moving data between compute nodes and storage, faces monumental challenges from new applications, memory, and storage architectures considered in the designs of exascale systems. As the storage hierarchy is expanding to include node-local persistent memory, burst buffers, etc., as well as disk-based storage, data movement among these layers must be efficient. Parallel I/O libraries of the future should be capable of handling file sizes of many terabytes and beyond. In this paper, we describe new capabilities we have developed in Hierarchical Data Format version 5 (HDF5), the most popular parallel I/O library for scientific applications. HDF5 is one of the most used libraries at the leadership computing facilities for performing parallel I/O on existing HPC systems. The state-of-the-art features we describe include: Virtual Object Layer (VOL), Data Elevator, asynchronous I/O, full-featured single-writer and multiple-reader (Full SWMR), and parallel querying. In this paper, we introduce these features, their implementations, and the performance and feature benefits to applications and other libraries.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Hierarchical Data Format for Nuclear Data Sensitivities

The SCALE code system includes capabilities for sensitivity and uncertainty (S/U) analysis as part of its TSUNAMI code suite. The sensitivity of a quantity of interest (for example, an application’s $k_{eff}$) to nuclear data is stored as a profile in a text-based file, which is known as a sensitivity data file (SDF). The sensitivity profile can be used to calculate uncertainties, correlation coefficients, and similarity indices. One of the goals of the present work was to seek general performance improvements in the TSUNAMI code suite, starting with the TSUNAMI-IP code for calculating similarity indices. Through profiling, it was found that reading the text-based sensitivity files was a performance bottleneck in the TSUNAMI-IP code. In a typical TSUNAMI-IP calculation, an application might be compared to thousands of benchmarks, thus requiring the reading of thousands of SDFs. Reading of binary-based data is generally faster than reading text-based data. Hierarchical Data Format 5 (HDF5) is a binary-based format that also benefits from being portable, and it can be inspected with nonproprietary tools. This paper describes an HDF5-based file format that has been introduced for SDFs.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2019

l3d2vtk: An MCNPTools Utility to Enable LNK3DNT File Visualization & Post-Processing

This paper describes recent work to develop and incorporate a utility program into the MCNPTools package that enables LNK3DNT file visualization & post-processing. A LNK3DNT file is a binary Cartesian, cylindrical, or spherical structured mesh file that describes material and material relative density distributions by mesh voxel. This mesh file format is principally used by the PARTISN discrete-ordinates radiation-transport code. However, the MCNP ® Monte Carlo radiation-transport code can use LNK3DNT files to specify geometry, and the MCNP code can produce LNK3DNT files representing a user-defined region of the MCNP constructive solid geometry (CSG). Because of the use of LNK3DNT files as MCNP input and output and with other computer codes, improving the ability to visualize and/or quantitatively post-process LNK3DNT files benefits end users. In particular, it is helpful to visually verify that the material and density distributions are as expected. Also, an analyst can orient viewers with respect to other in formation, e.g., by co-plotting a voxelized representation of MCNP CSG with mesh tally and/or particle track results. However, until now, the only capability to interactively interrogate LNK3DNT files is through a 2-D slice plotter provided by the Monte Carlo Application ToolKit (MCATK), which is not currently distributed outside of Los Alamos National Laboratory. The utility created as part of this work converts LNK3DNT binary files to an ASCII XML-based StructuredMesh VTK (.vts) file. The geometries supported are 1-D spherical and cylindrical ( r ), 1-D Cartesian ( x ), 2-D cylindrical ( r-z and r-θ ) and Cartesian ( x-y ), and 3-D cylindrical ( r-z-θ ) and Cartesian ( x-y-z ). This file format is chosen to minimize external dependencies (e.g., on an HDF5 library) and to provide a file that can be interrogated by the user in a text editor or standard visualization applications (e.g., ParaView or VisIt). However, other file formats may provide more utility such as XDMF, which can use HDF5 and is discussed further as something to explore as future work. This paper is organized as follows: a brief introduc tion to the MCNPTools library and utility programs is given, the LNK3DNT-to-VTK (l3d2vtk) utility functionality is de scribed, and then an example post-processing workflow using the ParaView application is given.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

Utah FORGE: Neubrex Well 16B(78)-32 DAS Data - April, 2024

This dataset comprises Distributed Acoustic Sensing (DAS) data collected from the Utah FORGE monitoring well 16B(78)-32 (the producer well) during hydraulic fracture stimulation operations conducted in April 2024. The data were acquired continuously over the stimulation period at a temporal sampling rate of 10,000 Hz (10 kS/s) and a spatial resolution of approximately 3.35 feet (1.02109 meters). The measurements were captured using a Neubrex NBX-S4100 Time Gated Digital DAS interrogator unit connected to a single-mode fiber optic cable, which was permanently installed within the casing string. All recorded channels correspond to downhole segments of the fiber optic cable, from a measured depth (MD) of 5,369.35 feet to 10,352.11 feet. The DAS data reflect raw acoustic energy generated by physical processes within and surrounding the well during stimulation activities at wells 16A(78)-32 and 16B(78)-32. These data have potential applications in analyzing cross-well strain, far-field strain rates (including microseismic activity), induced seismicity, and seismic imaging. Metadata embedded in the attributes of the HDF5 files include detailed information on the measured depths of the channels, interrogation parameters, and other acquisition details. The dataset also includes a recording of a seminar held on September 19, 2024, where Neubrex's Chief Operating Officer presented insights into the data collection, analysis, and preliminary findings. The raw data files, stored in HDF5 format, are organized chronologically according to the recording intervals from April 9 to April 24, 2024, with each file corresponding to a 12-second recording interval.

15 GEOTHERMAL ENERGY↗
DOE OSTI2022

Domain-Specific Type-Safe APIs for Hierarchical Scientific Data with Modern C++

General-purpose library application programming interfaces (APIs) for self-describing hierarchical scientific data storage, such as the HDF5 and NetCDF libraries, are traditionally of runtime nature. Runtime errors for entry existence and data types are typically caught later in the development process of higher-level application-specific APIs. In this paper, we propose exploiting modern C++ metaprogramming features to add compile-time type-safety to improve the interaction with a well-defined metadata-rich scientific schema in domain-specific hierarchical datasets. We tackle two aspects of common use: (i) direct data access, (ii) flexible “in-memory” index models for efficient search and data processing. The proposed APIs use C++17’s template type auto deduction features, C++11’s enum class for type-safety and C-style preprocessor macros for generative templated code. We showcase the pros and cons of our initial work on the standard NeXus schema used for annotating and storing experimental neutron scattering data at several facilities around the world on top of HDF5. Extendable compile-time type-safe APIs are a desirable feature that could be indexed by any modern integrated development environment (IDE). Hence, such APIs can help ease the learning curve for domain scientists using a less error-prone software interaction to enhance the findability of their data without resorting to a domain-specific language (DSL).

Godoy, William↗
DOE OSTI2012

Generalized Nuclear Data: A New Structure (with Supporting Infrastructure) for Handling Nuclear Data

The Evaluated Nuclear Data File (ENDF) format was designed in the 1960s to accommodate neutron reaction data to support nuclear engineering applications in power, national security and criticality safety. Over the years, the scope of the format has been extended to handle many other kinds of data including charged particle, decay, atomic, photo-nuclear and thermal neutron scattering. Although ENDF has wide acceptance and support for many data types, its limited support for correlated particle emission, limited numeric precision, and general lack of extensibility mean that the nuclear data community cannot take advantage of many emerging opportunities. More generally, the ENDF format provides an unfriendly environment that makes it difficult for new data evaluators and users to create and access nuclear data. The Cross Section Evaluation Working Group (CSEWG) has begun the design of a new Generalized Nuclear Data (or 'GND') structure, meant to replace older formats with a hierarchy that mirrors the underlying physics, and is aligned with modern coding and database practices. In support of this new structure, Lawrence Livermore National Laboratory (LLNL) has updated its nuclear data/reactions management package Fudge to handle GND structured nuclear data. Fudge provides tools for converting both the latest ENDF format (ENDF-6) and the LLNL Evaluated Nuclear Data Library (ENDL) format to and from GND, as well as for visualizing, modifying and processing (i.e., converting evaluated nuclear data into a form more suitable to transport codes) GND structured nuclear data. GND defines the structure needed for storing nuclear data evaluations and the type of data that needs to be stored. But unlike ENDF and ENDL, GND does not define how the data are to be stored in a file. Currently, Fudge writes the structured GND data to a file using the eXtensible Markup Language (XML), as it is ASCII based and can be viewed with any text editor. XML is a meta-language, meaning that it has a primitive set of definitions for representing hierarchical data/text in a file. Other meta-languages, like HDF5 which stores the data in binary form, can also be used to store GND in a file. In this paper, we will present an overview of the new GND data structures along with associated tools in Fudge.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2018

catsHTM: A Tool for Fast Accessing and Cross-matching Large Astronomical Catalogs

Fast access to large catalogs is required for some astronomical applications. Here we introduce the catsHTM tool, consisting of several large catalogs reformatted into HDF5-based file format, which can be downloaded and used locally. To allow fast access, the catalogs are partitioned into hierarchical triangular meshes and stored in HDF5 files. Several tools are provided to perform efficient cone searches at resolutions spanning from a few arc-seconds to degrees, within a few milliseconds time. The first released version includes the following catalogs (by alphabetical order): 2MASS, 2MASS extended sources, AKARI, APASS, Cosmos, DECaLS/DR5, FIRST, GAIA/DR1, GAIA/DR2, GALEX/DR6Plus7, HSC/v2, IPHAS/DR2, NED redshifts, NVSS, Pan-STARRS1/DR1, PTF photometric catalog, ROSAT faint source, SDSS sources, SDSS/DR14 spectroscopy, SkyMapper, Spitzer/SAGE, Spitzer/IRAC galactic center, UCAC4, UKIDSS/DR10, VST/ATLAS/DR3, VST/KiDS/DR3, WISE and XMM. We provide Python code that allows to perform cone searches, as well as MATLAB code for performing cone searches, catalog cross-matching, general searches, as well as load and create these catalogs.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2016

An Adaptable Seismic Data Format

We present ASDF, the Adaptable Seismic Data Format, a modern and practical data format for all branches of seismology and beyond. The growing volume of freely available data coupled with ever expanding computational power opens avenues to tackle larger and more complex problems. Current bottlenecks include inefficient resource usage and insufficient data organization. Properly scaling a problem requires the resolution of both these challenges, and existing data formats are no longer up to the task. ASDF stores any number of synthetic, processed or unaltered waveforms in a single file. A key improvement compared to existing formats is the inclusion of comprehensive meta information, such as event or station information, in the same file. Additionally, it is also usable for any non-waveform data, for example, cross-correlations, adjoint sources or receiver functions. Last but not least, full provenance information can be stored alongside each item of data, thereby enhancing reproducibility and accountability. Any data set in our proposed format is self-describing and can be readily exchanged with others, facilitating collaboration. The utilization of the HDF5 container format grants efficient and parallel I/O operations, integrated compression algorithms and check sums to guard against data corruption. To not reinvent the wheel and to build upon past developments, we use existing standards like QuakeML, StationXML, W3C PROV and HDF5 wherever feasible. Usability and tool support are crucial for any new format to gain acceptance. We developed mature C/Fortran and Python based APIs coupling ASDF to the widely used SPECFEM3D_GLOBE and ObsPy toolkits.

58 GEOSCIENCES↗
DOE OSTI2021

Characterizing Impacts of Storage Faults on HPC Applications: A methodology and insights

In recent years, the increasing complexity in scientific simulations and emerging demands for training heavy artificial intelligence models require massive and fast data accesses, which urges high-performance computing (HPC) platforms to equip with more advanced storage infrastructures such as solid-state disks (SSDs). While SSDs offer high-performance I/O, it remains unclear about the reliability challenges faced by the HPC applications under the SSD-related failures, in particular, failures resulting in data corruptions. The goal of this paper is to understand the impact of SSD-related data corruptions on the behaviors of complex HPC applications. To this end, we propose FFIS, a FUSE-based fault injection framework that systematically introduces storage faults into the application layer to model the errors originated from SSDs. FFIS is able to plant different I/O related faults into the data returned from underlying file systems, which also enables the investigation on the error resilience characteristics of the scientific file format for the first time. We demonstrate the use of FFIS with three representative real HPC applications, show how each application reacts to the data corruptions, and provide insights on the error resilience of the widely-adopted HDF5 file format for the HPC applications.

Fang, Bo↗