Engineering topics

Poole, Stephen W.

Publications and source records attributed to Poole, Stephen W..

8 records

DOE OSTI2019

Parallel storage system with burst buffer appliance for storage of partitioned key-value store across a plurality of storage tiers

Parallel storage systems are provided with a burst buffer appliance for storage of a partitioned key-value store across a plurality of storage tiers. The burst buffer appliance is configured to store key-value data on first and second storage tiers comprising respective disjoint subsets of storage. One or more servers in the partitioned key-value store provide (i) an interface between one or more local key-value stores and one or more additional storage architectures, and/or (ii) a key value interface that provides access to at least one hardware key-value store. The local key-values are optionally sorted both locally by the local store and globally by the overall MDHIM framework. The local and global sorting can be leveraged to provide batch input/output (IO) operations that aggregate multiple requests into batch operations on one or more sorted ranges.

Bent, John M.↗

DOE OSTI2019

Partitioned key-value store with one-sided communications for secondary global key lookup by range-knowledgeable clients

Partitioned key-value stores with one-sided communications are provided for secondary global key lookup by range-knowledgeable clients. A secondary global lookups is performed in a partitioned key-value store by receiving a request from an application for a secondary global key; accessing metadata to identify a first server storing the secondary global key; instructing the identified first server to store a first value corresponding to the requested secondary global key into a memory address of the client; and returning program control to the application, wherein the first server identifies a primary key corresponding to the requested secondary global key and instructs a second server storing the corresponding primary key to store a second value corresponding to the primary key into the memory address of the client. The second server stores the second value corresponding to the primary key into the first memory address of the client.

Bent, John M.↗

DOE OSTI2016

Parallel sort with a ranged, partitioned key-value store in a high perfomance computing environment

Improved sorting techniques are provided that perform a parallel sort using a ranged, partitioned key-value store in a high performance computing (HPC) environment. A plurality of input data files comprising unsorted key-value data in a partitioned key-value store are sorted. The partitioned key-value store comprises a range server for each of a plurality of ranges. Each input data file has an associated reader thread. Each reader thread reads the unsorted key-value data in the corresponding input data file and performs a local sort of the unsorted key-value data to generate sorted key-value data. A plurality of sorted, ranged subsets of each of the sorted key-value data are generated based on the plurality of ranges. Each sorted, ranged subset corresponds to a given one of the ranges and is provided to one of the range servers corresponding to the range of the sorted, ranged subset. Each range server sorts the received sorted, ranged subsets and provides a sorted range. A plurality of the sorted ranges are concatenated to obtain a globally sorted result.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2015

Parallel log structured file system collective buffering to achieve a compact representation of scientific and/or dimensional data

Collective buffering and data pattern solutions are provided for storage, retrieval, and/or analysis of data in a collective parallel processing environment. For example, a method can be provided for data storage in a collective parallel processing environment. The method comprises receiving data to be written for a plurality of collective processes within a collective parallel processing environment, extracting a data pattern for the data to be written for the plurality of collective processes, generating a representation describing the data pattern, and saving the data and the representation.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2015

Defense of Cyber Infrastructures Against Cyber-Physical Attacks Using Game-Theoretic Models

The operation of cyber infrastructures relies on both cyber and physical components, which are subject to incidental and intentional degradations of different kinds. Within the context of network and computing infrastructures, we study the strategic interactions between an attacker and a defender using game-theoretic models that take into account both cyber and physical components. The attacker and defender optimize their individual utilities expressed as sums of cost and system terms. First, we consider a Boolean attack-defense model, wherein the cyber and physical sub-infrastructures may be attacked and reinforced as individual units. Second, we consider a component attack-defense model wherein their components may be attacked and defended, and the infrastructure requires minimum numbers of both to function. We show that the Nash equilibrium under uniform costs in both cases is computable in polynomial time, and it provides high-level deterministic conditions for the infrastructure survival. When probabilities of successful attack and defense, and of incidental failures are incorporated into the models, the results favor the attacker but otherwise remain qualitatively similar. This approach has been motivated and validated by our experiences with UltraScience Net infrastructure, which was built to support high-performance network experiments. In conclusion, the analytical results, however, are more general, and we apply them to simplified models of cloud and high-performance computing infrastructures.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2014

Utility functions and resource management in an oversubscribed heterogeneous computing environment

We model an oversubscribed heterogeneous computing system where tasks arrive dynamically and a scheduler maps the tasks to machines for execution. The environment and workloads are based on those being investigated by the Extreme Scale Systems Center at Oak Ridge National Laboratory. Utility functions that are designed based on specifications from the system owner and users are used to create a metric for the performance of resource allocation heuristics. Each task has a time-varying utility (importance) that the enterprise will earn based on when the task successfully completes execution. We design multiple heuristics, which include a technique to drop low utility-earning tasks, to maximize the total utility that can be earned by completing tasks. The heuristics are evaluated using simulation experiments with two levels of oversubscription. The results show the benefit of having fast heuristics that account for the importance of a task and the heterogeneity of the environment when making allocation decisions in an oversubscribed environment. Furthermore, the ability to drop low utility-earning tasks allow the heuristics to tolerate the high oversubscription as well as earn significant utility.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2014

OpenSHMEM Reference Implementation using UCCS-uGNI Transport Layer. In: PGAS '14 Proceedings of the 8th International Conference on Partitioned Global Address Space Programming Models, Article No. 18

OpenSHMEM is a library interface implementation and specification that enables the implementation of the Partitioned Global Address Space (PGAS) model. It exports modern RDMA network functionality and communication semantics to applications very efficiently. There are many closed source implementations of OpenSHMEM for modern RDMA interconnects such as InfiniBand and Cray's Gemini and Aries. Given the important role that Cray systems play in HPC, in this paper, we present an open source implementation of OpenSHMEM for Cray XE/XK/XC systems. To implement OpenSHMEM, we use the uGNI interface. uGNI is a generic interface that is designed for multiple programming models. The interface fits well the goal of UCCS. Having OpenSHMEM with UCCS-uGNI allows usage of the same implementation over multiple interconnects. This also translates into many advantages that come with common code such as resource sharing, increasing productivity because of less code maintenance, etc. Preliminary results show that OpenSHMEM-UCCS performs comparable to state-of-the-art Cray SHMEM for Put, Get, and AMO operations.

Janjusic, Tomislav↗

DOE OSTI2013

Graph Generator Survey

The benchmarking effort within the Extreme Scale Systems Center at Oak Ridge National Laboratory seeks to provide High Performance Computing benchmarks and test suites of interest to the DoD sponsor. The work described in this report is a part of the effort focusing on graph generation. A previously developed benchmark, SystemBurn, allowed the emulation of different application behavior profiles within a single framework. To complement this effort, similar capabilities are desired for graph-centric problems. This report examines existing synthetic graph generator implementations in preparation for further study on the properties of their generated synthetic graphs.

analytics↗