Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Stores”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

New Total-Ionizing-Dose Resistant Data Storing Technique for NAND Flash Memory

This paper describes a new non-charge-based data storing technique in NAND flash memory called watermark that encodes read-only data in the form of physical properties of flash memory cells. Unlike traditional charge-based data storing method in flash memory, the proposed technique is resistant to total ionizing dose (TID) effects. To evaluate its resistance to irradiation effects, we analyze data stored in several commercial single-level-cell (SLC) flash memory chips from different vendors and technology nodes. These chips are irradiated using a Co-60 gamma-ray source array for up to 100 krad(Si) at Sandia National Laboratories. Experimental evaluation performed on a flash chip from Samsung shows that the intrinsic bit error rate (BER) of watermark increases from 0.8% for TID = 0 krad(Si) to 1% for TID = 100 krad(Si). Conversely, the BER of charge-based data stored on the same chip increases from 0% at TID = 0 krad(Si) to 1.5% at TID = 100 krad(Si). Overall, the results imply that the proposed technique may potentially offer significant improvements in data integrity relative to traditional charge-based data storage for very high radiation (TID > 100 krad(Si)) environments. These gains in data integrity relative to the charge-based data storage are useful in radiation-prone environments, but they come at the cost of increased write times and higher BERs before irradiation.

36 MATERIALS SCIENCE↗

A Ceph S3 Object Data Store for HEP

We present a novel data format design that obviates the need for data tiers by storing individual event data products in column objects. The objects are stored and retrieved through Ceph S3 technology, with a layout designed to minimize metadata volume and maximize data processing parallelism. Performance benchmarks of data storage and retrieval are presented.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DDStore: Distributed Data Store for Scalable Training of Graph Neural Networks on Large Atomistic Modeling Datasets

Graph neural networks (GNNs) are a class of Deep Learning models used in designing atomistic materials for effective screening of large chemical spaces. To ensure robust prediction, GNN models must be trained on large volumes of atomistic data on leadership class supercomputers. Even with the advent of modern architectures that consist of multiple storage layers that include node-local NVMe devices in addition to device memory for caching large datasets, extreme-scale model training faces I/O challenges at scale.We present DDStore, an in-memory distributed data store designed for GNN training on large-scale graph data. DDStore provides a hierarchical, distributed, data caching technique that combines data chunking, replication, low-latency random access, and high throughput communication. DDStore achieves near-linear scaling for training a GNN model using up to 1000 GPUs on the Summit and Perlmutter supercomputers, and reaches up to a 6.15x reduction in GNN training time compared to state-of-the-art methodologies.

Choi, Jong Youl↗

Summary of Graphite Data Stored within NDMAS

The Graphite Technology Development Project provides data to support the design of graphite core components within specific reactor service conditions of the next generation of high-temperature, gas-cooled nuclear reactors. Physical, mechanical, and thermal properties of nuclear grade graphite were characterized for specimens that were unirradiated, irradiated, and irradiated under various stress conditions. The material properties include diameter, length, mass, density, compressive strength, tensile strength, flexural strength, modulus, resistivity, thermal diffusivity, and thermal expansion coefficient. Baseline graphite specimens are unirradiated from different grades (2114, IG-110, NBG-17, NBG-18, and PCEA) and different types used in different characterization tests (compressive, flexural, tensile, one-inch cylinder, and quarter-inch cylinder). The Advanced Graphite Creep (AGC) irradiation specimens are from a much larger number of grades and cylinder types (creep, piggyback, and pencil). For baseline graphite, characterization data for 7,756 specimens extracted from thirteen graphite billets were captured to the NDMAS database. For AGC experiments, four irradiation campaigns have been completed: AGC-1, AGC-2, AGC-3, and AGC-4. The ongoing HDG-1 (High-Dose Graphite) experiment, which began irradiation with Cycle 168B on August 26, 2020, includes specimens previously irradiated in AGC-2 in addition to the specimens originally destined for AGC-5.. Besides the characterization data, the AGC data includes irradiation monitoring and physics data representing the irradiation conditions of AGC specimens. Currently, all data for AGC-1, AGC-2, and AGC-3 have been captured to the NDMAS database. Only AGC-4 pre-irradiation and irradiation monitoring data have been captured, and HDG-1 pre-irradiation data are in process of being captured for unirradiated specimens. To date, a total of 38,149 material property records have been captured into NDMAS database for baseline and AGC specimens. All characterization data are qualified for use according to their perspective data verification reports. For the AGC irradiation campaigns, a total of 41,502 qualified physics calculation records were added for AGC-1, AGC-2, and AGC-3. Finally, a total of 173,698,505 AGC irradiation monitoring data records (thermocouple temperature, gas flow rate, gas pressure, gas moisture, applied load, and specimen displacement) have been captured to NDMAS; the majority of those records (~94%) are qualified data records.

11 - NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Extreme Compression for Large Scale Data Store

For the last 5 years Accelogic pioneered and perfected a radically new theory of numerical computing codenamed “Compressive Computing”, which has an extremely profound impact on real-world computer science [1]. At the core of this new theory is the discovery of one of its fundamental theorems which states that, under very general conditions, the vast majority (typically between 70% and 80%) of the bits used in modern large-scale numerical computations are absolutely irrelevant for the accuracy of the end result. This theory of Compressive Computing provides mechanisms able to identify (with high intelligence and surgical accuracy) the number of bits (i.e., the precision) that can be used to represent numbers without affecting the substance of the end results, as they are computed and vary in real time. The bottom line outcome would be to provide a state-of-the-art compression algorithm that surpasses those currently available in the ROOT framework, with the purpose of enabling substantial economic and operational gains (including speedup) for High Energy and Nuclear Physics data storage/analysis. In our initial studies, a factor of nearly x4 (3.9) compression was achieved with RHIC/STAR data where ROOT compression managed only x1.4.In this contribution, we will present our concepts of “functionally lossless compression”, have a glance at examples and achievements in other communities, present the results and outcome of our current, ongoing R&D, as well as present a high-level view of our plan to move forward with a ROOT implementation that would deliver a basic solution readily integrated into HENP applications. As a collaboration of experimental scientists, private industry, and the ROOT Team, our aim is to capitalize on the substantial success delivered by the initial effort and produce a robust technology properly packaged as an open-source tool that could be used by virtually every experiment around the world as means for improving data management and accessibility.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Techniques for storing data to enhance recovery and detection of data corruption errors

Often there are errors when reading data from computer memory. To detect and correct these errors, there are multiple types of error correction codes. Disclosed is an error correction architecture that creates a codeword having a data portion and an error correction code portion. Swizzling rearranges the order of bits and distributes the bits among different codewords. Because the data is redistributed, a potential memory error of up to N contiguous bits, where N for example equals 2 times the number of codewords swizzled together, only affects up to, at most, two bits per swizzled codeword. This keeps the error within the error detecting capabilities of the error correction architecture. Furthermore, this can allow improved error correction and detection without requiring a change to error correcting code generators and checkers.

Mills, Peter↗

Techniques for storing data to enhance recovery and detection of data corruption errors

Often there are errors when reading data from computer memory. To detect and correct these errors, there are multiple types of error correction codes. Disclosed is an error correction architecture that creates a codeword having a data portion and an error correction code portion. Swizzling rearranges the order of bits and distributes the bits among different codewords. Because the data is redistributed, a potential memory error of up to N contiguous bits, where N for example equals 2 times the number of codewords swizzled together, only affects up to, at most, two bits per swizzled codeword. This keeps the error within the error detecting capabilities of the error correction architecture. Furthermore, this can allow improved error correction and detection without requiring a change to error correcting code generators and checkers.

Mills, Peter↗

Electricity Baseline 2021 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2021 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilizes the appdirs Python dependency (https://pypi.org/project/appdirs/). An overview of the ElectricityLCI data stores may be found on the README (https://github.com/USEPA/ElectricityLCI/blob/v2.0/README.md#data-store). This submission includes the background data used to generate the 2021 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: python -c "import appdirs; print(appdirs.user_data_dir())"). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2021 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; LCI; Life Cycle; data inventory↗

Electricity Baseline 2020 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2020 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilizes the appdirs Python dependency (https://pypi.org/project/appdirs/). An overview of the ElectricityLCI data stores may be found on the README (https://github.com/USEPA/ElectricityLCI/blob/v2.0/README.md#data-store). This submission includes the background data used to generate the 2020 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: python -c "import appdirs; print(appdirs.user_data_dir())"). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2020 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; LCI; Life Cycle; data inventory↗

Electricity Baseline 2022 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2022 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilized the appdirs Python dependency (https://pypi.org/project/appdirs/). This submission includes the background data used to generate the 2022 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: `python -c "import appdirs; print(appdirs.user_data_dir())"`). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2022 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; data inventory↗

funcX: Federated Function as a Service for Science

Here, funcX is a distributed function as a service (FaaS) platform that enables flexible, scalable, and high performance remote function execution. Unlike centralized FaaS systems, funcX decouples the cloud-hosted management functionality from the edge-hosted execution functionality. funcX's endpoint software can be deployed, by users or administrators, on arbitrary laptops, clouds, clusters, and supercomputers, in effect turning them into function serving systems. funcX's cloud-hosted service provides a single location for registering, sharing, and managing both functions and endpoints. It allows for transparent, secure, and reliable function execution across the federated ecosystem of endpoints-enabling users to route functions to endpoints based on specific needs. funcX uses containers (e.g., Docker, Singularity, and Shifter) to provide common execution environments across endpoints. funcX implements various container management strategies to execute functions with high performance and efficiency on diverse funcX endpoints. funcX also integrates with an in-memory data store and Globus for managing data that may span endpoints. We motivate the need for funcX, present our prototype design and implementation, and demonstrate, via experiments on two supercomputers, that funcX can scale to more than 130000 concurrent workers. We show that funcX's container warming-aware routing algorithm can reduce the completion time for 3,000 functions by up to 61% compared to a randomized algorithm and the in-memory data store can speed up data transfers by up to 3x compared to a shared file system.

97 MATHEMATICS AND COMPUTING↗

The Design and Implementation of a Secure Datastore Based on Ethereum Smart Contract

In this paper, we present a secure datastore based on an Ethereum smart contract. Our research is guided by three research questions. First, we will explore to what extend a smart-contract-based datastore should resemble a traditional database system. Second, we will investigate how to store the data in a smart-contract-based datastore for maximum flexibility while minimizing the gas consumption. Third, we seek answers regarding whether or not a smart-contract-based datastore should incorporate complex processing such as data encryption and data analytic algorithms. The proposed smart-contract-based datastore aims to strike a good balance between several constraints: (1) smart contracts are publicly visible, which may create a confidentiality concern for the data stored in the datastore; (2) unlike traditional database systems, the Ethereum smart contract programming language (i.e., Solidity) offers very limited data structures for data management; (3) all operations that mutate the blockchain state would incur financial costs and the developers for smart contracts must make sure sufficient gas is provisioned for every smart contract call, and ideally, the gas consumption should be minimized. Our investigation shows that although it is essential for a smart-contract-based datastore to offer some basic data query functionality, it is impractical to offer query flexibility that resembles that of a traditional database system. Furthermore, we propose that data should be structured as tag-value pairs, where the tag serves as a non-unique key that describes the nature of the value. We also conclude that complex processing should not be allowed in the smart contract due to the financial burden and security concerns. The tag-based secure datastore designed this way also defines its applicative perimeter, i.e., only applications that align with our strategy would find the proposed datastore a good fit. Those that would rather incur higher financial cost for more data query flexibility and/or less user burden on data pre- and post-processing would find the proposed database too restrictive.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Automated Indexing of Structured Scientific Metadata Using Apache Solr

Scientific datasets are continuously growing with the amount of raw data being collected worldwide. This amount of data poses the biggest challenge to web search engines on how to retrieve them efficiently. This paper discusses how major scientific data centers are using popular open-source search platforms such as Solr [1] to retrieve structured data stored in data sources such as relational database management systems using its import handler mechanisms [2]. Additionally, we will also focus on how we can configure Solr to serve advanced full-text, faceted search capabilities, along with its key features, which simplify representing and delivering better performance to the scientific search interfaces.

Guntupally, Kavya↗

Stability Analysis of Inline ZFP Compression for Floating-Point Data in Iterative Methods

Currently, the dominating constraint in many high performance computing applications is data capacity and bandwidth, in both internode communications and even moreso in intranode data motion. A new approach to address this limitation is to make use of data compression in the form of a compressed data array. Storing data in a compressed data array and converting to standard IEEE-754 types as needed during a computation can reduce the pressure on bandwidth and storage. However, repeated conversions (lossy compression and decompression) introduce additional approximation errors, which need to be shown to not significantly affect the simulation results. Here, we extend recent work that analyzed the error of a single use of compression and decompression of the ZFP compressed data array representation to the case of time-stepping and iterative schemes, where an advancement operator is repeatedly applied in addition to the conversions. We show that the accumulated error for iterative methods involving fixed-point and time evolving iterations is bounded under standard constraints. An upper bound is established on the number of additional iterations required for the convergence of stationary fixed-point iterations. An additional analysis of traditional forward and backward error of stationary iterative methods using ZFP compressed arrays is also presented. The results of several 1D, 2D, and 3D test problems are provided to demonstrate the correctness of the theoretical bounds.

97 MATHEMATICS AND COMPUTING↗

Electricity Baseline 2022

The Electricity Baseline (2022) is a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data and was created using the ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0). The Python package used the "ELCI_2022" model configuration to set the facility and generation data sources and years that were used to create this life cycle inventory, which were taken from publicly accessible datasets and automatically curated into a local data store. An archive of the data stores used in this model is available online: https://doi.org/10.18141/2569193. This model is presented in GreenDelta's openLCA schema v2 JSON-LD format (https://greendelta.github.io/olca-schema/).

Electricity; LCA; data inventory↗

Electricity Baseline 2020

The Electricity Baseline (2020) is a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data and was created using the ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0). The Python package used the "ELCI_2020" model configuration to set the facility and generation data sources and years that were used to create this life cycle inventory, which were taken from publicly accessible datasets and automatically curated into a local data store. An archive of the data stores used in this model is available online: https://doi.org/10.18141/2569605. This model is presented in GreenDelta's openLCA schema v2 JSON-LD format (https://greendelta.github.io/olca-schema/).

Electricity; LCA; LCI; data inventory↗

Electricity Baseline 2021

The Electricity Baseline (2021) is a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data and was created using the ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0). The Python package used the "ELCI_2021" model configuration to set the facility and generation data sources and years that were used to create this life cycle inventory, which were taken from publicly accessible datasets and automatically curated into a local data store. An archive of the data stores used in this model is available online: https://doi.org/10.18141/2569576. This model is presented in GreenDelta's openLCA schema v2 JSON-LD format (https://greendelta.github.io/olca-schema/).

Electricity; LCA; LCI; Life Cycle↗