SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
H5Z-ZFP is a compression plugin (filter) for the HDF5 library based upon the ZFP-0.5.0 compression library. It supports 4- or 8-byte integer or floating point HDF5 datasets of any dimension but partitioned in 1, 2, or 3 dimensional chunks. It supports ZFP's four fundamental modes of operation; rate, precision, accuracy or expert. It is a lossy compression plugin.
Practitioners of lattice QCD/QFT have been some of the primary pioneer users of the state-of-the-art high-performance-computing systems, and contribute towards the stress tests of such new machines as soon as they become available. As with all aspects of high-performance-computing, I/O is becoming an increasingly specialized component of these systems. In order to take advantage of the latest available high-performance I/O infrastructure, to ensure reliability and backwards compatibility of data files, and to help unify the data structures used in lattice codes, we have incorporated parallel HDF5 I/O into the SciDAC supported USQCD software stack. Here we present the design and implementation of this I/O framework. Our HDF5 implementation outperforms optimized QIO at the 10-20% level and leaves room for further improvement by utilizing appropriate dataset chunking.
In High Energy Physics (HEP), experimentalists generate large volumes of data that, when analyzed, helps us better understand the fundamental particles and their interactions. This data is often captured in many files of small size, creating a data management challenge for scientists. In order to better facilitate data management, transfer, and analysis on large scale platforms, it is advantageous to aggregate data further into a smaller number of larger files. However, this translation process can consume significant time and resources, and if performed incorrectly the resulting aggregated files can be inefficient for highly parallel access during analysis on large scale platforms. In this paper, we present our case study on parallel I/O strategies and HDF5 features for reducing data aggregation time, making effective use of compression, and ensuring efficient access to the resulting data during analysis at scale. We focus on NOvA detector data in this case study, a large-scale HEP experiment generating many terabytes of data. Here, the lessons learned from our case study inform the handling of similar datasets, thus expanding community knowledge related to this common data management task.
There are many commercial ultrasonic tomography devices (UTDs) available for use in nondestructive evaluation (NDE) of reinforced concrete structures. These devices emit, measure, and store ultrasonic signals typically in the 25 kHz to 5 MHz frequency range. UTDs are characterized by a composition of multiple transducers, also known as a transducer array or phased array. Often, UTDs data are in a proprietary format. Consequently, NDE research data is limited to those who have prior non-disclosure agreements or the appropriate licenses. Thus, there is a need for a proper universal data framework to exist such that proprietary file datasets for different concrete specimens can be converted, organized, and stored with relative metadata for individual or collaborative NDE research. Building upon the Hierarchical Data Format (HDF5) model, we have developed a UTD data management framework and Graphic User Interface (GUI) to promote the algorithmic reconstruction of ultrasonic data in a controlled environment for easily reproducible and publishable results.
For in transit processing, one of the fundamental challenges is the efficient movement of data from producers to consumers. Exploiting the flexibility offered by the SENSEI generic in situ framework, we have developed a number of different in transit data transport mechanisms. In this work, we focus on the transport mechanism that leverages the HDF5 parallel I/O library, and investigate the performance characteristics of this transport mechanism. For in transit use cases at scale on HPC platforms, one might expect that an in transit data transport mechanism that uses faster layers of the storage hierarchy, such as DRAM memory, would always outperform a transport that uses slower layers of the storage hierarchy, such as an NVRAM-based persistent storage presented as a distributed file system. However, our test results show that the performance of the transport using NVRAM is competitive with the transport that uses socket-based data movement across varying levels of producer and consumer concurrency.}, booktitle = {Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization
An example of reading the Galacticus HDF5 file in R and a demonstration of some of the features of the data.
We have built a new framework that enables independent metadata updates within HDF5 files. Our framework includes the following components: an overlay network that's optimized for 1-to-All and All-to-All communication; root-less/server-less Bcast and AllReduce operations for reduced synchronization; a distributed consensus-making platform; an execution order keeping system, and a progress engine for generic threadless concurrence handling.
H5bench benchmark suite contains a list of applications that are used to measure the HDF5 I/O performance from various aspects.
H5bench benchmark suite contains a list of applications that are used to measure the HDF5 I/O performance from various aspects.
For several years now, Fermilab has been investigating the use of HDF5 for large-scale analysis of experimental high energy physics (HEP) data.
This paper describes an approach to interrogating the fission matrix available in the HDF5-formatted restart file (also known as the “runtape” file), which is produced by the upcoming public release of the MCNP R code, version 6.3. The fission matrix, its eigenvalues, and its eigenvectors, are important tools for characterizing fissile systems such as research and power reactors and for accelerating the convergence of the Monte Carlo fission-source site distribution.
This report describes the verification performed to ensure that a new HDF5-based MCNP® unstructured mesh (UM) [1] file correctly represents elemental values on the associated mesh geometry.
A Python tool for converting the MCNP6 HDF5 PTRAC file to a list of Python trees is presented. The particle trees store MCNP6 simulated events for each history using parent-child relationships, which ensures that branching processes are accurately reproduced. A variety of post-processing scripts are presented and used in conjunction with the Python particle trees to make special tallies that are currently not available in the MCNP6 software and visualize the particle tracks.
An example of reading the Galacticus HDF5 file in R and a demonstration of some of the features of the data.
Explore the source record for details and available documents.
Not Available
The current runtape in MCNP is a Fortran unformatted IO dump of approximately 800 variables used for restarting an MCNP simulation. This format has a number of technical issues for those who interact with it. First, Fortran unformatted IO is not guaranteed to be portable. As a result, a runtape generated on one machine is not necessarily readable on another. Second, for users who wish to extract data from the runtape, there is no simple approach. It is impossible to extract one variable without knowing the size and position of every other one. Finally, the runtape itself is not backwards compatible such that a future version of MCNP could open it. Users are expected to keep around versions of MCNP that are compatible with saved runtapes.