Search NASA⌕ Search

Engineering topics

Grider, Gary

Publications and source records attributed to Grider, Gary.

At least 19 records

Pattern-aware prefetching using parallel log-structured file system

Techniques are provided for pattern-aware prefetching using a parallel log-structured file system. At least a portion of one or more files is accessed by detecting at least one pattern in a non-sequential access of the one or more files; and obtaining at least a portion of the one or more files based on the detected at least one pattern. The obtaining step comprises, for example, a prefetching or pre-allocation of the at least the portion of the one or more files. A prefetch cache can store the portion of the one or more obtained files. The cached portion of the one or more files can be provided from the prefetch cache to an application requesting the at least a portion of the one or more files.

Bent, John M.↗

Streaming Data Reorganization at Scale with DeltaFS Indexed Massive Directories

We report complex storage stacks providing data compression, indexing, and analytics help leverage the massive amounts of data generated today to derive insights. It is challenging to perform this computation, however, while fully utilizing the underlying storage media. This is because, while storage servers with large core counts are widely available, single-core performance and memory bandwidth per core grow slower than the core count per die. Computational storage offers a promising solution to this problem by utilizing dedicated compute resources along the storage processing path. We present DeltaFS Indexed Massive Directories (IMDs), a new approach to computational storage. DeltaFS IMDs harvest available (i.e., not dedicated) compute, memory, and network resources on the compute nodes of an application to perform computation on data. We demonstrate the efficiency of DeltaFS IMDs by using them to dynamically reorganize the output of a real-world simulation application across 131,072 CPU cores. DeltaFS IMDs speed up reads by 1,740x while only slightly slowing down the writing of data during simulation I/O for in situ data processing.

97 MATHEMATICS AND COMPUTING↗

Method and system for reducing metadata overhead in a two-tier storage architecture

Example embodiments of the present invention relate to a method, a system, and a computer program product for maintaining dynamically loadable namespaces. The method includes presenting, to an application, a file system having a global single namespace (SN) comprising a plurality of dynamically-loadable namespaces (DLNs) maintained in a capacity tier (CT) of object storage of a storage system. In response to a job submitted by the application, dynamically loading a first dynamically-loadable namespace, including the metadata only for the first dynamically-loadable namespace, associated with the job from the capacity tier of object storage into a fast tier (FT) of storage of the storage system.

97 MATHEMATICS AND COMPUTING↗

Metadata compression

A computer-executable method, system, and computer program product for managing a Parallel-Logged File System (PLFS) on a data storage system, wherein the data storage system is in communication with an application that updates the data storage system with checkpoint data, the computer-executable method, system, and computer program product comprising receiving a first checkpoint including data from the application; analyzing the data in the first checkpoint to determine whether a pattern exists in the checkpoint data; and upon a positive determination, adding the pattern into PLFS.

97 MATHEMATICS AND COMPUTING↗

Method and system for semantic metadata compression in a two-tier storage system using copy-on-write

Example embodiments of the present invention relate to a method, a system, and a computer program product for managing metadata. The method includes determining a common characteristic of metadata entries for a plurality of files in a directory of a file system and computing a representative metadata entry for the plurality of metadata entries according to the common characteristic of the metadata entries. The plurality of metadata entries in the file system then may be represented according to the computed representative metadata entry.

97 MATHEMATICS AND COMPUTING↗

Parallel storage system with burst buffer appliance for storage of partitioned key-value store across a plurality of storage tiers

Parallel storage systems are provided with a burst buffer appliance for storage of a partitioned key-value store across a plurality of storage tiers. The burst buffer appliance is configured to store key-value data on first and second storage tiers comprising respective disjoint subsets of storage. One or more servers in the partitioned key-value store provide (i) an interface between one or more local key-value stores and one or more additional storage architectures, and/or (ii) a key value interface that provides access to at least one hardware key-value store. The local key-values are optionally sorted both locally by the local store and globally by the overall MDHIM framework. The local and global sorting can be leveraged to provide batch input/output (IO) operations that aggregate multiple requests into batch operations on one or more sorted ranges.

Bent, John M.↗

Storage Systems and Input/Output: Organizing, Storing, and Accessing Data for Scientific Discovery. Report for the DOE ASCR Workshop on Storage Systems and I/O. [Summary Brief]

In September, 2018, the Department of Energy, Office of Science, Advanced Scientific Computing Research Program convened a workshop to identify key challenges and define research directions that will advance the field of storage systems and I/O over the next 5-7 years.The workshop concluded that addressing these combined challenges and opportunities requires tools and techniques that greatly extend traditional approaches and require new research directions. Key research opportunities were identified. This summary brief highlights three areas: The Impact of Storage and I/O, Research Opportunities, and, A new and productive Storage and I/O Environment. The full report from the workshop can be viewed at DOI: 10.2172/1491994.

97 MATHEMATICS AND COMPUTING↗

Partitioned key-value store with one-sided communications for secondary global key lookup by range-knowledgeable clients

Partitioned key-value stores with one-sided communications are provided for secondary global key lookup by range-knowledgeable clients. A secondary global lookups is performed in a partitioned key-value store by receiving a request from an application for a secondary global key; accessing metadata to identify a first server storing the secondary global key; instructing the identified first server to store a first value corresponding to the requested secondary global key into a memory address of the client; and returning program control to the application, wherein the first server identifies a primary key corresponding to the requested secondary global key and instructs a second server storing the corresponding primary key to store a second value corresponding to the primary key into the memory address of the client. The second server stores the second value corresponding to the primary key into the first memory address of the client.

Bent, John M.↗

Global distributed file append using log-structured file system

Techniques are provided for globally appending data from a group of distributed processes to a shared file using a log-structured file system. Data generated by a plurality of processes in a parallel computing system are appended to a shared file by storing the data to the shared file using a log-structured file system (such as a Parallel Log-Structured File System (PLFS)); and generating an index entry for the data, the index entry comprising a logical offset entry and a timestamp entry indicating a time of the storage, wherein the logical offset entry is resolved at read time. The logical offset entry can be populated with an append placeholder that is resolved when the shared file is read. At read time, a plurality of the index entries associated with the shared file can be sorted using the timestamp entry to deliver the requested shared file to a requesting application.

97 MATHEMATICS AND COMPUTING↗

Distributed metadata servers in a file system with separate metadata servers for file metadata and directory metadata

A file system is provided having a plurality of distributed metadata servers with separate metadata servers for file metadata and directory metadata. An exemplary distributed metadata storage system comprises a plurality of compute nodes, wherein at least one compute node comprises a file system client that generates a directory metadata request and/or a file metadata request; a plurality of directory metadata servers to serve directory metadata in response to a given directory metadata request; and a plurality of file metadata servers to serve file metadata in response to a given file metadata request. The plurality of directory metadata servers are distinct from the plurality of file metadata servers. The compute nodes optionally comprise an object client that requests object data from an object store system. The number of directory metadata servers and the number of file metadata servers are optionally determined independently to balance for different workloads, and can be dynamically readjusted for evolving workloads.

97 MATHEMATICS AND COMPUTING↗

Computing erasure metadata and data layout prior to storage using a processing platform

Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Fail-Slow at Scale: Evidence of Hardware Performance Faults in Large Production Systems

Fail-slow hardware is an under-studied failure mode. We present a study of 114 reports of fail-slow hardware incidents, collected from large-scale cluster deployments in 14 institutions. We show that all hardware types such as disk, SSD, CPU, memory, and network components can exhibit performance faults. We made several important observations such as faults convert from one form to another, the cascading root causes and impacts can be long, and fail-slow faults can have varying symptoms. From this study, we make suggestions to vendors, operators, and systems designers.

fail-slow↗

Storage Systems and Input/Output: Organizing, Storing, and Accessing Data for Scientific Discovery. Report for the DOE ASCR Workshop on Storage Systems and I/O. [Full Workshop Report]

In September, 2018, the Department of Energy, Office of Science, Advanced Scientific Computing Research Program convened a workshop to identify key challenges and define research directions that will advance the field of storage systems and I/O over the next 5–7 years. The workshop concluded that addressing these combined challenges and opportunities requires tools and techniques that greatly extend traditional approaches and require new research directions. Key research opportunities were identified.

97 MATHEMATICS AND COMPUTING↗

Hardware Evaluation: Abstract Machine Models and Proxy Architectures for Exascale Computing

The strategic and economic impacts of high performance computing (HPC) systems, over the last several decades, have enabled dramatic improvements in manufacturing, design, development, and research, across almost every sector of the U.S. economy, including genetic analysis for agriculture and medicine, oil exploration, electronics, aircraft, and automotive design, and of course national defense technologies. HPC has repeatedly compounded reductions in time-to-solution and time-to-market in each of these areas. In recognition of these strategic and economic impacts, the Department of Energy (DOE) created the Exascale Computing Project (ECP) as a broad-based technology project focusing on simultaneously co-evolving high-performance computing architecture, system software, and application software. While many details of the exascale architectures are undefined, communicating the form and direction of this co-evolution of advances, in each of these three components, across the project, presents a need for a common language through which these advances can be shared. In this document we present a series of abstract representations designed to allow application developers to focus on the aspects of the machine that are important or relevant to performance and code structure. These abstract machine models (AMMs) describe the proposed architectures at the component and system level and are intended as communication aids between application developers and hardware architects during the co-design process. In addition to providing a common language for communication, the constraints of protecting the intellectual property of ECP’s vendor partners requires that the architectural advances be presented in an abstract sense, rather than exposing the specifics of an individual vendor’s designs, while still providing enough detail such that application developers will be able to reason about the performance trade-offs in the design space of their applications.

97 MATHEMATICS AND COMPUTING↗

Key-value store with internal key-value storage interface

A key-value store is provided having one or more key-value storage interfaces. A key-value store on at least one compute node comprises a memory for storing a plurality of key-value pairs; and an abstract storage interface comprising a software interface module that communicates with at least one persistent storage device providing a key-value interface for persistent storage of one or more of the plurality of key-value pairs, wherein the software interface module provides the one or more key-value pairs to the at least one persistent storage device in a key-value format. The abstract storage interface optionally processes one or more batch operations on the plurality of key-value pairs. A distributed embodiment for a partitioned key-value store is also provided.

97 MATHEMATICS AND COMPUTING↗

Distributed metadata servers for cluster file systems using shared low latency persistent key-value metadata store

A cluster file system is provided having a plurality of distributed metadata servers with shared access to one or more shared low latency persistent key-value metadata stores. A metadata server comprises an abstract storage interface comprising a software interface module that communicates with at least one shared persistent key-value metadata store providing a key-value interface for persistent storage of key-value metadata. The software interface module provides the key-value metadata to the at least one shared persistent key-value metadata store in a key-value format. The shared persistent key-value metadata store is accessed by a plurality of metadata servers. A metadata request can be processed by a given metadata server independently of other metadata servers in the cluster file system. A distributed metadata storage environment is also disclosed that comprises a plurality of metadata servers having an abstract storage interface to at least one shared persistent key-value metadata store.

97 MATHEMATICS AND COMPUTING↗

Storage of sparse files using parallel log-structured file system

A sparse file is stored without holes by storing a data portion of the sparse file using a parallel log-structured file system; and generating an index entry for the data portion, the index entry comprising a logical offset, physical offset and length of the data portion. The holes can be restored to the sparse file upon a reading of the sparse file. The data portion can be stored at a logical end of the sparse file. Additional storage efficiency can optionally be achieved by (i) detecting a write pattern for a plurality of the data portions and generating a single patterned index entry for the plurality of the patterned data portions; and/or (ii) storing the patterned index entries for a plurality of the sparse files in a single directory, wherein each entry in the single directory comprises an identifier of a corresponding sparse file.

97 MATHEMATICS AND COMPUTING↗