SEARCH · Search NASA

Results for “parallel file systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

All sources NASA NTRS DOE OSTI

DOE OSTI2020

Characterizing Output Bottlenecks of a Production Supercomputer: Analysis and Implications

This article studies the I/O write behaviors of the Titan supercomputer and its Lustre parallel file stores under production load. The results can inform the design, deployment, and configuration of file systems along with the design of I/O software in the application, operating system, and adaptive I/O libraries.We propose a statistical benchmarking methodology to measure write performance across I/O configurations, hardware settings, and system conditions. Moreover, we introduce two relative measures to quantify the write-performance behaviors of hardware components under production load. In addition to designing experiments and benchmarking on Titan, we verify the experimental results on one real application and one real application I/O kernel, XGC and HACC IO, respectively. These two are representative and widely used to address the typical I/O behaviors of applications.In summary, we find that Titan’s I/O system is variable across the machine at fine time scales. This variability has two major implications. First, stragglers lessen the benefit of coupled I/O parallelism (striping). Peak median output bandwidths are obtained with parallel writes to many independent files, with no striping or write sharing of files across clients (compute nodes). I/O parallelism is most effective when the application—or its I/O libraries—distributes the I/O load so that each target stores files for multiple clients and each client writes files on multiple targets in a balanced way with minimal contention. Second, our results suggest that the potential benefit of dynamic adaptation is limited. In particular, it is not fruitful to attempt to identify “good locations” in the machine or in the file system: component performance is driven by transient load conditions and past performance is not a useful predictor of future performance. For example, we do not observe diurnal load patterns that are predictable.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Understanding Lustre Internals. Second Edition

The Lustre file system has become a preferred storage resource for systems on the Top500 list, and it is often the file system of choice for small- to medium-sized HPC systems that require parallel shared access to data. Several resources exist to help users deploy and configure Lustre, but the same cannot be said for resources that explain the inner workings of the Lustre source code. A previous ORNL technical report entitled "Understanding Lustre Filesystem Internals" (ORNL/TM-2009/117) provided an excellent summary of Lustre subsystem operations. However, that report is over a decade old and is based on Lustre version 1.6. Since that report was published, Lustre has evolved significantly. Several subsystems underwent significant code changes and many new features have been added to the file system, bringing the current Lustre version up to 2.15.This report aims to document and explain the internal workings of the latest version of the Lustre file system. It will provide more complete and up-to-date information than the previous technical report and should serve as a foundational document for anyone interested in Lustre software development. Key data structures will be described along with the APIs used for interaction among the various Lustre subsystems. Although the Lustre software is constantly being developed, the details in this document should remain relevant for the forseeable future.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey

Growing interest in Artificial Intelligence (AI) has resulted in a surge in demand for faster methods of Machine Learning (ML) model training and inference. This demand for speed has prompted the use of high performance computing (HPC) systems that excel in managing distributed workloads. Because data is the main fuel for AI applications, the performance of the storage and I/O subsystem of HPC systems is critical. In the past, HPC applications accessed large portions of data written by simulations or experiments or ingested data for visualizations or analysis tasks. ML workloads perform small reads spread across a large number of random files. This shift of I/O access patterns poses several challenges to modern parallel storage systems. In this paper, we survey I/O in ML applications on HPC systems, and target literature within a 6-year time window from 2019 to 2024. We define the scope of the survey, provide an overview of the common phases of ML, review available profilers and benchmarks, examine the I/O patterns encountered during offline data preparation, training, and inference, and explore I/O optimizations utilized in modern ML frameworks and proposed in recent literature. Lastly, we seek to expose research gaps that could spawn further R&D.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

lustre_fssync

lustre_fssync is a tool used to sync an entire Lustre file system from one cluster to another. It uses Lustre user quotas, parallel I/O utilities and the SLURM resource manager to balance and synchronize all contents between systems.

Harr, CameronM↗

DOE OSTI2021

HOSS!

The Hall-D Online Skim System (HOSS) was developed to simultaneously solve two issues for the high intensity GlueX experiment. One was to parallelize the writing of raw data files to disk in order to improve bandwidth. The other was to distribute the raw data across multiple compute nodes in order to produce calibration skims of the data online. The highly configurable system employs RDMA, RAM disks, and zeroMQ driven by Python to simultaneously store and process the full high intensity GlueX data stream.

Lawrence, David↗

DOE OSTI2018

Calibrated simulation analysis for integration of evaporative cooling and radiant cooling system for different Indian climatic zones

Radiant cooling system (RCS) has proven to be an energy efficient system for meeting a building's cooling requirements. RCS is energy efficient and provides better thermal comfort compared to conventional all-air heating ventilation air conditioning (HVAC) system. To further improve the efficiency of RCS, a parallel evaporative cooling system (cooling tower) is coupled with RCS and analyzed for different climatic condition using calibrated model. A statistical analysis of weather files (based on wet bulb temperature), was used to identify the availability of useful water for cooling tower integration with RCS. A comprehensive simulation feasibility study of the application of cooling tower in RCS was performed for different cities to cover every climatic zones of India. It was found that in summer, the wet bulb temperature (WBT) of different climatic zones, except warm-humid, is suitable for the integration of cooling tower with RCS. An experimental setup was designed and developed for integration of parallel chiller and cooling tower with panel-based RCS for cooling and a dedicated outdoor air system (DOAS) for dehumidification and ventilation. Experiments were conducted for chiller and cooling tower operated RCS in Composite climate of Jaipur, Rajasthan, India. Building Energy Models (BEM) were developed for both the chiller-operated RCS and cooling tower-operated RCS in EnergyPlus and calibrated with the measured data. Using the calibrated models, performance of the system was analyzed for different climatic zones of India. A chiller-operated RCS was considered as a baseline to compare the annual energy saving potential and monthly performance of the cooling tower integrated RCS. Here, in the cooling tower–operated RCS, a total annual savings of 7% in hot and dry climates, 11% in composite climates and 20% in temperate climates was achieved compared to the chiller-operated RCS.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2020

Characterization and identification of HPC applications at leadership computing facility

High Performance Computing (HPC) is an important method for scientific discovery via large-scale simulation, data analysis, or artificial intelligence. Leadership-class supercomputers are expensive, but essential to run large HPC applications. The Petascale era of supercomputers began in 2008, with the first machines achieving performance in excess of one petaflops, and with the advent of new supercomputers in 2021 (e.g., Aurora, Frontier), the Exascale era will soon begin. However, the high theoretical computing capability (i.e., peak FLOPS) of a machine is not the only meaningful target when designing a supercomputer, as the resources demand of applications varies. A deep understanding of the characterization of applications that run on a leadership supercomputer is one of the most important ways for planning its design, development and operation. In order to improve our understanding of HPC applications, user demands and resource usage characteristics, we perform correlative analysis of various logs for different subsystems of a leadership supercomputer. This analysis reveals surprising, sometimes counter-intuitive patterns, which, in some cases, conflicts with existing assumptions, and have important implications for future system designs as well as supercomputer operations. For example, our analysis shows that while the applications spend significant time on MPI, most applications spend very little time on file I/O. Combined analysis of hardware event logs and task failure logs show that the probability of a hardware FATAL event causing task failure is low. Combined analysis of control system logs and file I/O logs reveals that pure POSIX I/O is used more widely than higher level parallel I/O. Based on holistic insights of the application gained through combined and co-analysis of multiple logs from different perspectives and general intuition, we engineer features to "fingerprint" HPC applications. We use t-SNE (a machine learning technique for dimensionality reduction) to validate the explainability of our features and finally train machine learning models to identify HPC applications or group those with similar characteristic. To the best of our knowledge, this is the first work that combines logs on file I/O, computing, and inter-node communication for insightful analysis of HPC applications in production.

Liu, Zhengchun↗

DOE OSTI2021

Online data analysis and reduction: An important co-design motif for extreme-scale computers

A growing disparity between supercomputer computation speeds and I/O rates means that it is rapidly becoming infeasible to analyze supercomputer application output only after that output has been written to a file system. Instead, data-generating applications must run concurrently with data reduction and/or analysis operations, with which they exchange information via high-speed methods such as interprocess communications. The resulting parallel computing motif, online data analysis and reduction (ODAR), has important implications for both application and HPC systems design. Here we introduce the ODAR motif and its co-design concerns, describe a co-design process for identifying and addressing those concerns, present tools that assist in the co-design process, and present case studies to illustrate the use of the process and tools in practical settings.

Data Analysis↗

DOE OSTI2023

LCIO

Performance of file systems shift during their life cycles. Evaluating this performance change over time is not trivial. Complexity arises in the interplay between external (i.e. application I/O workloads) and internal (i.e. the filesystem state) factors. Many benchmarks can test how a filesystem performs at the current snapshot state, but to observe the change over time necessitates that the filesystem state mutate (age) between benchmark runs. For a large-scale HPC parallel filesystem, the sheer scale and amount of interacting components during I/O operations magnify these challenges. LCIO addresses the question - how will the filesystem perform at different stages of its life cycle? LCIO is a synthetic benchmark, which provides the file system aging process to increase the amount of information that existing benchmarks like IOR and MDTest yield, as well as provide additional points of data that will be useful to system architects and engineers.

Bachstein, Matthew↗

DOE OSTI2019

Production Application Performance Data Streaming for System Monitoring

In this article, we present an approach to streaming collection of application performance data. Practical application performance tuning and troubleshooting in production high-performance computing (HPC) environments requires an understanding of how applications interact with the platform, including (but not limited to) parallel programming libraries such as Message Passing Interface (MPI). Several profiling and tracing tools exist that collect heavy runtime data traces either in memory (released only at application exit) or on a file system (imposing an I/O load that may interfere with the performance being measured). Although these approaches are beneficial in development stages and post-run analysis, a systemwide and low-overhead method is required to monitor deployed applications continuously. This method must be able to collect information at both the application and system levels to yield a complete performance picture. In our approach, an application profiler collects application event counters. A sampler uses an efficient inter-process communication method to periodically extract the application counters and stream them into an infrastructure for performance data collection. We implement a tool-set based on our approach and integrate it with the Lightweight Distributed Metric Service (LDMS) system, a monitoring system used on large-scale computational platforms. LDMS provides the infrastructure to create and gather streams of performance data in a low overhead manner. We demonstrate our approach using applications implemented with MPI, as it is one of the most common standards for the development of large-scale scientific applications. We utilize our tool-set to study the impact of our approach on an open source HPC application, Nalu. Our tool-set enables us to efficiently identify patterns in the behavior of the application without source-level knowledge. We leverage LDMS to collect system-level performance data and explore the correlation between the system and application events. Finally, we demonstrate how our tool-set can help detect anomalies with a low latency. We run tests on two different architectures: a system enabled with Intel Xeon Phi and another system equipped with Intel Xeon processor. Our overhead study shows our method imposes at most 0.5% CPU usage overhead on the application in realistic deployment scenarios.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2023

Online Monitoring of Medium Voltage Cable Systems with Spread Spectrum Time Domain and Frequency Domain Reflectometry

In-service failures of wave energy convertor (WEC) cable systems can have a significant cost and power availability impact. Close parallel research 2019 data showed > 1B£ and 9 Terra-Watt-Hours associated with global off-shore wind (OSW) cable failures (Strang-Moran 2020). OSW is a closely related technology but currently is significantly cheaper than WEC technology. For wave energy to compete, the problem of reliable cable transmission must be mitigated. This project develops isolation technology to allow online high frequency reflectometry testing of medium voltage cables (1 to 10 kV and higher) without arcing or damage to the test instrument. Online spread spectrum time domain reflectometry (SSTDR) testing has been established for low voltage cable systems in the aircraft and rail industry and the ability to detect and locate cable flaws of interest is well understood. Extending reflectometry testing to medium voltage systems could enable detection of cable damage before failures occur thereby allowing repair and replacement of damaged cable segments to be scheduled and managed. The seedling project succeeded to pass and receive high frequency SSTDR signals onto a cable up to 1 kV using a parallel trace isolation circuit board that can be connected onto the test cable. The approach used a novel circuit design for which an invention disclosure has been filed. A proposed sapling project would extend the technology toward the higher operating voltages used by WEC systems, thereby enabling online SSTDR cable monitoring. The goal of the seedling project was to extend the capability of the ARENA cable/motor test bed to address medium voltages and to develop a high pass filter isolation architecture to protect the reflectometry instrument from the low frequency (DC – 60 Hz) line voltage while allowing the high frequency diagnostic signal to pass to and from the test instrument to the live line. Initial efforts focused on passive LCR filter circuits to reduce 60 Hz levels below 10 volts from a 10 kV line while allowing the MHz high frequency chirps to pass onto the cables and for mV signals to be detected. We discovered that the parasitic loss behavior of real high voltage components precluded this approach from working. An alternate approach was adapted for the electric field to couple between two parallel traces on a printed circuit board much like a radio-frequency coupler. The challenge here was and is to have the parallel traces close enough to each other to effectively pass the high frequency chirp onto the live line and receive any reflected signal from any encountered impedance change along the cable. This reflected signal will be in the mV range. The traces however must be far enough apart to not allow arcing on the board. A design with 3 mm spacing was determined to allow the high frequency signal to pass onto the live line and receive the mV signal back into the instrument while reducing the 60 Hz voltage amplitude by >80 dB (more than a factor of 10,000) without allowing arcing from across the parallel traces. This was confirmed by simulation and test.

16 TIDAL AND WAVE POWER↗

DOE OSTI2021

Adaptive Spatially Aware I/O for Multiresolution Particle Data Layouts

Large-scale simulations on nonuniform particle distributions that evolve over time are widely used in cosmology, molecular dynamics, and engineering. Such data are often saved in an unstructured format that neither preserves spatial locality nor provides metadata for accelerating spatial or attribute subset queries, leading to poor performance of visualization tasks. Furthermore, the parallel I/O strategy used typically writes a file per process or a single shared file, neither of which is portable or scalable across different HPC systems. We present a portable technique for scalable, spatially aware adaptive aggregation that preserves spatial locality in the output. We evaluate our approach on two supercomputers, Stampede2 and Summit, and demonstrate that it outperforms prior approaches at scale, achieving up to 2.5× faster writes and reads for nonuniform distributions. Furthermore, the layout written by our method is directly suitable for visual analytics, supporting low-latency reads and attribute-based filtering with little overhead.

Usher, Will↗

DOE OSTI2020

ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems

Scientific applications at exascale generate and analyze massive amounts of data. A critical requirement of these applications is the capability to access and manage this data efficiently on exascale systems. As such, parallel I/O, the key technology enables moving data between compute nodes and storage, faces monumental challenges from new applications, memory, and storage architectures considered in the designs of exascale systems. As the storage hierarchy is expanding to include node-local persistent memory, burst buffers, etc., as well as disk-based storage, data movement among these layers must be efficient. Parallel I/O libraries of the future should be capable of handling file sizes of many terabytes and beyond. In this paper, we describe new capabilities we have developed in Hierarchical Data Format version 5 (HDF5), the most popular parallel I/O library for scientific applications. HDF5 is one of the most used libraries at the leadership computing facilities for performing parallel I/O on existing HPC systems. The state-of-the-art features we describe include: Virtual Object Layer (VOL), Data Elevator, asynchronous I/O, full-featured single-writer and multiple-reader (Full SWMR), and parallel querying. In this paper, we introduce these features, their implementations, and the performance and feature benefits to applications and other libraries.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Scalable filesystem enumeration and metadata operations

Systems, apparatus, and methods are disclosed for performing scalable operations in a file system, including POSIX-like file systems. Metadata entries in a namespace or directory tree are sharded across multiple file metadata servers. An enumeration operation, such as listing a directory, is parallelized across the multiple file metadata servers, while retaining standard functionality transparently to clients. Other enumeration operations include no-output operations such as changing file attributes or deleting a file, and cumulative operations such as counting total disk space usage. The parallelization is compatible with tree-level parallelization and storage-level parallelization. Disclosed technologies can be applied to other fields requiring scalable enumeration, such as database and network applications.

Grider, Gary A.↗

DOE OSTI2020

How fast can one resize a distributed file system?

Efficient resource utilization becomes a major concern as large-scale distributed computing infrastructures keep growing in size. Malleability, the possibility for resource managers to dynamically increase or decrease the amount of resources allocated to a job, is a promising way to save energy and costs. However, state-of-the-art parallel and distributed storage systems have not been designed with malleability in mind. The reason is mainly the supposedly high cost of data transfers required by resizing operations. Nevertheless, as network and storage technologies evolve, old assumptions about potential bottlenecks can be revisited. In this study, we evaluate the viability of malleability as a design principle for a distributed storage system. We specifically model the minimal duration of the commission and decommission operations. To show how our models can be used in practice, we evaluate the performance of these operations in HDFS, a relevant state-of-the-art distributed file system. We show that the existing decommission mechanism of HDFS is good when the network is the bottleneck, but can be accelerated by up to a factor 3 when storage is the limiting factor. We also show that the commission in HDFS can be substantially accelerated. With the highlights provided by our model, we suggest improvements to speed both operations in HDFS. We discuss how the proposed models can be generalized for distributed file systems with different assumptions and what perspectives are open for the design of efficient malleable distributed file systems.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Scalable augmented enumeration and metadata operations for large filesystems

Systems, apparatus, and methods are disclosed for performing scalable operations in a file system. Metadata entries in a namespace or directory tree are sharded across multiple file metadata servers. An augmented enumeration operation, such as listing a directory, is parallelized across the multiple file metadata servers, transparently to clients. Exemplary augmentation features can include filtering and sorting. Augmentation features can be executed concurrently with enumeration, prior to enumeration, after enumeration, or as a combination of these, and can utilize pre-built index structures or holding structures for intermediate results. Augmented enumeration operations can also include no-output operations such as changing file attributes or deleting a file, and cumulative operations such as counting total disk space usage. The parallelization is compatible with tree-level parallelization and storage-level parallelization. Disclosed technologies can be applied to other fields requiring scalable enumeration, such as database and network applications.

Grider, Gary A.↗

DOE OSTI2021

Improving I/O Performance for Exascale Applications through Online Data Layout Reorganization

The applications being developed within the U.S. Exascale Computing Project (ECP) to run on imminent Exascale computers will generate scientific results with unprecedented fidelity and record turn-around time. Many of these codes are based on particle-mesh methods and use advanced algorithms, especially dynamic load-balancing and mesh-refinement, to achieve high performance on Exascale machines. Yet, as such algorithms improve parallel application efficiency, they raise new challenges for I/O logic due to their irregular and dynamic data distributions. Thus, while the enormous data rates of Exascale simulations already challenge existing file system write strategies, the need for efficient read and processing of generated data introduces additional constraints on the data layout strategies that can be used when writing data to secondary storage. We review these I/O challenges and introduce two online data layout reorganization approaches for achieving good tradeoffs between read and write performance. We demonstrate the benefits of using these two approaches for the ECP particle-in-cell simulation WarpX, which serves as a motif for a large class of important Exascale applications. Here, we show that by understanding application I/O patterns and carefully designing data layouts we can increase read performance by more than 80 percent.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Evolution of the ROOT Tree I/O

The ROOT TTree data format encodes hundreds of petabytes of High Energy and Nuclear Physics events. Its columnar layout drives rapid analyses, as only those parts (“branches”) that are really used in a given analysis need to be read from storage. Its unique feature is the seamless C++ integration, which allows users to directly store their event classes without explicitly defining data schemas. In this contribution, we present the status and plans of the future ROOT 7 event I/O. Along with the ROOT 7 interface modernization, we aim for robust, where possible compile-time safe C++ interfaces to read and write event data. On the performance side, we show first benchmarks using ROOT’s new experimental I/O subsystem that combines the best of TTrees with recent advances in columnar data formats. A core ingredient is a strong separation of the high-level logical data layout (C++ classes) from the low-level physical data layout (storage backed nested vectors of simple types). We show how the new, optimized physical data layout speeds up serialization and deserialization and facilitates parallel, vectorized and bulk operations. This lets ROOT I/O run optimally on the upcoming ultra-fast NVRAM storage devices, as well as file-less storage systems such as object stores.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗