Search NASA⌕ Search

Engineering topics

Latham, Rob

Publications and source records attributed to Latham, Rob.

MPICH

MPICH is a high-performance and widely portable implementation of the MPI-4 standard from the Argonne National Laboratory. This release has all MPI 4 functions and features required by the standard with the exception of support for the user-defined data representations for I/O.

ECP↗

I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis

Using parallel file systems efficiently is a tricky problem due to inter-dependencies among multiple layers of I/O software, including high-level I/O libraries (HDF5, netCDF, etc.), MPI-IO, POSIX, and file systems (GPFS, Lustre, etc.). Profiling tools such as Darshan collect traces to help understand the I/O performance behavior. However, there are significant gaps in analyzing the collected traces and then applying tuning options offered by various layers of I/O software. Seeking to connect the dots between I/O bottleneck detection and tuning, we propose DXT Explorer, an interactive log analysis tool. In this paper, we present a case study using our interactive log analysis tool to identify and apply various I/O optimizations. We report an evaluation of performance improvement achieved for four I/O kernels extracted from science applications.

Bez, Jean Luca↗

mochi-quintain

mochi-quintain is a microservice (i.e., a Mochi provider) that responds to parameterized RPCs to generate synthetic concurrent server load.

Carns, Phil↗

mochi-hpc/mochi-margo

Margo provides Argobots-aware bindings to the Mercury RPC library. Margo combines Mercury and Argobots to simplify development of distributed services. Mercury operations are presented as conventional blocking operations, and RPC handlers are presented as sequential threads. This configuration enables high degree of concurrency while hiding the complexity associated with asynchronous communication progress and callback management.

Carns, Phil↗

SSG (Scalable Service Groups)

SSG is a group membership microservice based on the Mercury RPC system. It provides mechanisms for bootstrapping sets of Mercury processes into logical groups and for managing the membership of these process groups over time. At a high-level, each group collectively maintains a group view, which is just a mapping from group member identifiers to Mercury address information. The initial group membership view is specified completely when the group is bootstrapped (created).

Carns, Phil↗

Mobject

Mobject is a prototype Mochi object storage system based on RADOS.

Dorier, Matthieu↗

abt-io

abt-io provides Argobots-aware wrappers to common POSIX I/O functions.

Carns, Phil↗

Bake

Bake is a microservice (i.e., Mochi provider) for high performance bulk storage of raw data regions. Bake uses modular backends to store data on persistent memory, conventional file systems, or other storage media.

Carns, Phil↗

Thallium

Thallium is a C++14 library wrapping Margo, Mercury, and Argobots and providing an object-oriented way to use these libraries.

Dorier, Matthieu↗

Collective I/O Tuning Using Analytical and Machine-Learning Models

The ever larger demand of scientific applications for computation and data is currently driving a continuous increase in scale of parallel computers. The inherent complexity of scaling up a computing systems in terms of both hardware and software stack exposes an increasing number of factors impacting the performance and complicating the process of optimization. In particular, the optimization of parallel I/O has become increasingly challenging due to increasing storage hierarchy and well known performance variability of shared storage systems. This paper focuses on model-based autotuning of the two-phase collective I/O algorithm from a popular MPI distribution on the Blue Gene/Q architecture. We propose a novel hybrid model, constructed as a composition of analytical models for communication and storage operations and black-box models for the performance of the individual operations. We perform an in-depth study of the complexity involved in performance modeling including architecture, software stack and noise. In particular we address this challenges of modeling the performance of shared storage systems by building a benchmark that helps synthesizing factors such as topology, file caching, and noise. The experimental results show that the hybrid approach produces significantly better results than state-of-the-art machine learning approaches and shows a higher robustness to noise, at the cost of a higher modeling complexity

I/O performance modeling↗

ParNCL and ParGAL: Data-parallel Tools for Postprocessing of Large-scale Earth Science Data

Earth science high-performance applications often require extensive analysis of their output in order to complete the scien- tific goals or produce a visual image or animation. Often this analysis cannot be done in situ because it requires calculating time-series statistics from state sampled over the entire length of the run or analyzing the relationship between similar time series from previous simulations or observations. Many of the tools used for this postprocessing are not themselves high- performance applications, but the new Parallel Gridded Analysis Library (ParGAL) provides high-performance data-parallel versions of several common analysis algorithms for data from a structured or unstructured grid simulation. The library builds on several scalable systems, including the Mesh Oriented DataBase (MOAB), a library for representing mesh data that sup- ports structured, unstructured finite element, and polyhedral grids; the Parallel-NetCDF (PNetCDF) library; and Intrepid, an extensible library for computing operators (such as gradient, curl, and divergence) acting on discretized fields. We have used ParGAL to implement a parallel version of the NCAR Command Language (NCL) a scripting language widely used in the climate community for analysis and visualization. The data-parallel algorithms in ParGAL/ParNCL are both higher performing and more flexible than their serial counterparts.

97 MATHEMATICS AND COMPUTING↗

ISABELA for effective in situ compression of scientific data: ISABELA FOR EFFECTIVE IN-SITU REDUCTION OF SPATIO-TEMPORAL DATA

Exploding dataset sizes from extreme-scale scientific simulations necessitates efficient data management and reduction schemes to mitigate I/O costs. With the discrepancy between I/O bandwidth and computational power, scientists are forced to capture data infrequently, thereby making data collection an inherently lossy process. Although data compression can be an effective solution, the random nature of real-valued scientific datasets renders lossless compression routines ineffective. These techniques also impose significant overhead during decompression, making them unsuitable for data analysis and visualization, which require repeated data access. To address this problem, we propose an effective method for In situ Sort-And-B-spline Error-bounded Lossy Abatement (ISABELA) of scientific data that is widely regarded as effectively incompressible. With ISABELA, we apply a pre-conditioner to seemingly random and noisy data along spatial resolution to achieve an accurate fitting model that guarantees a ≥0.99 correlation with the original data. We further take advantage of temporal patterns in scientific data to compress data b ≈ 85%, while introducing only a negligible overhead on simulations in terms of runtime. ISABELA significantly outperforms existing lossy compression methods, such as wavelet compression, in terms of data reduction and accuracy. We extend upon our previous paper by additionally building a communication-free, scalable parallel storage framework on top of ISABELA-compressed data that is ideally suited for extreme-scale analytical processing. The basis for our storage framework is an inherently local decompression method (it need not decode the entire data), which allows for random access decompression and low-overhead task division that can be exploited over heterogeneous architectures. Furthermore, analytical operations such as correlation and query processing run quickly and accurately over data in the compressed space. Copyright © 2012 John Wiley & Sons, Ltd.

Computer Science↗