Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Stores”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2019

Analysis Ready Data in Analytics Optimized Data Stores for Analysis of Big Earth Data in the Cloud

Cloud computing offers the possibility of making the analysis of Big Data approachable for a wider community due to affordable access to computing power, an ecosystem of usable tools for parallel processing, and migration of many large datasets to archives in the cloud, allowing data-proximal computing. Generally, data analysis acceleration in the cloud comes from running multiple nodes in a split-combine-apply strategy. Data systems such as the Earth Observing System Data and Information System are in a position to "pre-split" the data by storing them in a data store that is optimized for data parallel computing, i.e., an Analytics-Optimized Data Store (AODS). A variety of approaches to AODS are possible, from highly scalable databases to scalable filesystems to data formats optimized for cloud access (e.g., zarr and cloud-optimized datasets), with the optimal choice dependent on both the types of analysis and the geospatial structure of the data. A key question is how much preprocessing of the data to do, both before splitting and as the first part of the apply step. Again, the geospatial structure of the data and the analysis type influence the decision, with the added complexity of the user type. Trans-disciplinary users who are not well-versed in the nuances of quality-filtering and georeferencing of remote sensing orbit/swath/scene data tend to ask for more highly processed data, relying on the data provider to make sensible decisions on preprocessing parameters. (This accounts for the popularity of "Level 3" gridded data, despite the lower spatial resolution it provides.) In this case, data can be preprocessed before the split, resulting in higher performance in the rest of the "apply" step, which can be transformative for use cases such as interactive data exploration at scale. Discipline researchers who are experienced with remote sensing data often prefer more flexibility in customizing the preprocessing data into Analysis Ready Data, resulting in more need for on-the-fly preprocessing.

Lynnes, Christopher↗
DOE OSTI2022

New Total-Ionizing-Dose Resistant Data Storing Technique for NAND Flash Memory

This paper describes a new non-charge-based data storing technique in NAND flash memory called watermark that encodes read-only data in the form of physical properties of flash memory cells. Unlike traditional charge-based data storing method in flash memory, the proposed technique is resistant to total ionizing dose (TID) effects. To evaluate its resistance to irradiation effects, we analyze data stored in several commercial single-level-cell (SLC) flash memory chips from different vendors and technology nodes. These chips are irradiated using a Co-60 gamma-ray source array for up to 100 krad(Si) at Sandia National Laboratories. Experimental evaluation performed on a flash chip from Samsung shows that the intrinsic bit error rate (BER) of watermark increases from 0.8% for TID = 0 krad(Si) to 1% for TID = 100 krad(Si). Conversely, the BER of charge-based data stored on the same chip increases from 0% at TID = 0 krad(Si) to 1.5% at TID = 100 krad(Si). Overall, the results imply that the proposed technique may potentially offer significant improvements in data integrity relative to traditional charge-based data storage for very high radiation (TID > 100 krad(Si)) environments. These gains in data integrity relative to the charge-based data storage are useful in radiation-prone environments, but they come at the cost of increased write times and higher BERs before irradiation.

36 MATERIALS SCIENCE↗
NASA NTRS1995

Alternate physical formats for storing data in HDF

Since its inception HDF has evolved to meet new demands by the scientific community to support new kinds of data and data structures, larger data sets, and larger numbers of data sets. The first generation of HDF supported simple objects and simple storage schemes. These objects were used to build more complex objects such as raster images and scientific data sets. The second generation of HDF provided alternate methods of storing data elements, making it possible to do such things as store extendible objects with in HDF, to store data externally from HDF files, and support data compression effectively. As we look to the next generation of HDF, we are considering fundamental changes to HDF, including a redefinition of the basic HDF object from a simple object to a more general, higher-level scientific data object that has certain characteristics, such as dimensionality, a more general atomic number type, and attributes. These changes suggest corresponding changes to the HDF file format itself.

Folk, Mike↗
NASA NTRSDate not supplied

High Performance Access to Archival Data Stored in HDF4 and HDF5 on Cloud Object Stores Without Reformatting the Files

Cloud computing offers numerous advantages for users of extensive Earth science data collections. These benefits encompass direct online access to data files and granules from any location, scalable access supporting parallel computing workflows, and flexible computing tools enabling innovative experimentation with processing techniques. However, older archival file formats designed for distinct computing systems hinder efficient access to decade-long time-series data when compared to data stored in modern cloud-optimized formats like Web Object Stores (WOS), exemplified by Amazon Web Services’ Simple Storage Service (S3). We describe DMR++ (Dataset Metadata Response plus plus), a technology facilitating efficient access to HDF5 (Hierarchical Data Format, version 5) and HDF4 files stored on WOS systems without requiring data reformatting. DMR++ achieves performance comparable to technologies like Zarr while preserving the original file structure, a substantial benefit considering the vast quantity of archival files held by organizations such as NASA. Moreover, DMR++ typically outperforms cloud-optimized versions of HDF5. Essentially an XML (Extensible Markup Language) document usually stored alongside the described data, DMR++ can also be generated on-the-fly but is generally created during data staging to the WOS. Archival files that use HDF4/5 often store large arrays of numerical data. The data in these files is often compressed, typically reducing their size by a factor of four or more. To achieve efficient access to portions of those arrays, they are 'chunked' into smaller sub-arrays, each individually compressed. The chunk size is a compromise, where spinning disks can efficiently access data in smaller chunks while S3 favors larger chunks. A simple optimization of aggregating smaller chunks that are stored adjacently, transferring them in a single access and then individually decompressing them will improve performance. NASA data pose an additional challenge: special Application Programmer Interface (API) libraries are often needed to compute some variables. These libraries are incompatible with WOS environments. Our solution involves storing computed values in the DMR++ document or a companion file, making them accessible like other variables and eliminating the need for specialized APIs. We outline specific optimizations for both satellite grid and swath data stored in HDF4-EOS2 (Earth Observing System).

James Gallagher↗
DOE OSTI2023

A Ceph S3 Object Data Store for HEP

We present a novel data format design that obviates the need for data tiers by storing individual event data products in column objects. The objects are stored and retrieved through Ceph S3 technology, with a layout designed to minimize metadata volume and maximize data processing parallelism. Performance benchmarks of data storage and retrieval are presented.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2023

DDStore: Distributed Data Store for Scalable Training of Graph Neural Networks on Large Atomistic Modeling Datasets

Graph neural networks (GNNs) are a class of Deep Learning models used in designing atomistic materials for effective screening of large chemical spaces. To ensure robust prediction, GNN models must be trained on large volumes of atomistic data on leadership class supercomputers. Even with the advent of modern architectures that consist of multiple storage layers that include node-local NVMe devices in addition to device memory for caching large datasets, extreme-scale model training faces I/O challenges at scale.We present DDStore, an in-memory distributed data store designed for GNN training on large-scale graph data. DDStore provides a hierarchical, distributed, data caching technique that combines data chunking, replication, low-latency random access, and high throughput communication. DDStore achieves near-linear scaling for training a GNN model using up to 1000 GPUs on the Summit and Perlmutter supercomputers, and reaches up to a 6.15x reduction in GNN training time compared to state-of-the-art methodologies.

Choi, Jong Youl↗
DOE OSTI2025

Summary of Graphite Data Stored within NDMAS

The Graphite Technology Development Project provides data to support the design of graphite core components within specific reactor service conditions of the next generation of high-temperature, gas-cooled nuclear reactors. Physical, mechanical, and thermal properties of nuclear grade graphite were characterized for specimens that were unirradiated, irradiated, and irradiated under various stress conditions. The material properties include diameter, length, mass, density, compressive strength, tensile strength, flexural strength, modulus, resistivity, thermal diffusivity, and thermal expansion coefficient. Baseline graphite specimens are unirradiated from different grades (2114, IG-110, NBG-17, NBG-18, and PCEA) and different types used in different characterization tests (compressive, flexural, tensile, one-inch cylinder, and quarter-inch cylinder). The Advanced Graphite Creep (AGC) irradiation specimens are from a much larger number of grades and cylinder types (creep, piggyback, and pencil). For baseline graphite, characterization data for 7,756 specimens extracted from thirteen graphite billets were captured to the NDMAS database. For AGC experiments, four irradiation campaigns have been completed: AGC-1, AGC-2, AGC-3, and AGC-4. The ongoing HDG-1 (High-Dose Graphite) experiment, which began irradiation with Cycle 168B on August 26, 2020, includes specimens previously irradiated in AGC-2 in addition to the specimens originally destined for AGC-5.. Besides the characterization data, the AGC data includes irradiation monitoring and physics data representing the irradiation conditions of AGC specimens. Currently, all data for AGC-1, AGC-2, and AGC-3 have been captured to the NDMAS database. Only AGC-4 pre-irradiation and irradiation monitoring data have been captured, and HDG-1 pre-irradiation data are in process of being captured for unirradiated specimens. To date, a total of 38,149 material property records have been captured into NDMAS database for baseline and AGC specimens. All characterization data are qualified for use according to their perspective data verification reports. For the AGC irradiation campaigns, a total of 41,502 qualified physics calculation records were added for AGC-1, AGC-2, and AGC-3. Finally, a total of 173,698,505 AGC irradiation monitoring data records (thermocouple temperature, gas flow rate, gas pressure, gas moisture, applied load, and specimen displacement) have been captured to NDMAS; the majority of those records (~94%) are qualified data records.

11 - NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
NASA NTRS1995

Storing Data In Circulating Electrons

Proposed memory device contains electrons traveling in circular orbit in vacuum. Orientations of electron spins (up or down) represents binary ones or zeros, respectively. Assuming length of each memory element 100 times classical fictitious electron radius, device theoretically capable of storing 446 gigabytes of data per meter of circumference of orbit.

Eastman, Ted R.↗
NASA NTRS2018

Accessing Data Stored in Amazon S3 Using the Hyrax OPeNDAP Server

For three years we have been investigating data storage and retrieval in the Amazon Cloud, striving to optimize both storage structures and (cache-enhanced retrieval processes. These optimizations are hidden from data users, who simply employ the Data Access Protocol (DAP), a popular, Web-based Application Programmer Interface (API) developed by OPeNDAP. We present our collected findings and their realization in the Hyrax data server. Specific techniques include caching data on spinning disk; accessing sharded data in place; optimizing the organizational structures for data within the flat key-value space of Simple Storage Service (S3); and employing an API extension that permits simultaneous operations on many datasets in a single request.

Data access↗
NASA NTRS2019

Cloud Giovanni: Reining in Costs and Improving Performance with Analytical Data Stores Using Scalable Serverless Architecture

Giovanni is the Geospatial Interactive Online Visualization ANd aNalysis Infrastructure developed at NASA GES DISC which provides a simple and intuitive way to visualize, analyze, and access vast amounts of Earth science data. It receives large number of user requests each day for a variety of analysis and visualization services, which leads to the big data challenge of serving gradually increasing large data volumes with diverse statistical algorithms. We hereby propose a multi-dimensional accumulation method which provides fast and cost-efficient cloud analysis for diverse services including both area averaging and time averaging. This method involves the weighted volume integration over multiple variable dimensions (time and space), and is implemented in AWS using Athena providing serverless and highly scalable data analysis. Compared to the standard method, this approach dramatically reduced the computational time by order of magnitude with a minimal AWS cost incurred. For example, for a benchmark of 10-year area averaging over the 1x1 degree daily variable, the computational time was reduced from minutes to seconds, and the Athena cost is only $5 for 100,000 requests.

Zhang, Hailiang↗
NASA NTRS1985

Storing Data and Video on One Tape

Microprocessor-based system originally developed for anthropometric research merges digital data with video images for storage on video cassette recorder. Combined signals later retrieved and displayed simultaneously on television monitor. System also extracts digital portion of stored information and transfers it to solid-state memory.

Nixon, J. H.↗
DOE OSTI2020

Extreme Compression for Large Scale Data Store

For the last 5 years Accelogic pioneered and perfected a radically new theory of numerical computing codenamed “Compressive Computing”, which has an extremely profound impact on real-world computer science [1]. At the core of this new theory is the discovery of one of its fundamental theorems which states that, under very general conditions, the vast majority (typically between 70% and 80%) of the bits used in modern large-scale numerical computations are absolutely irrelevant for the accuracy of the end result. This theory of Compressive Computing provides mechanisms able to identify (with high intelligence and surgical accuracy) the number of bits (i.e., the precision) that can be used to represent numbers without affecting the substance of the end results, as they are computed and vary in real time. The bottom line outcome would be to provide a state-of-the-art compression algorithm that surpasses those currently available in the ROOT framework, with the purpose of enabling substantial economic and operational gains (including speedup) for High Energy and Nuclear Physics data storage/analysis. In our initial studies, a factor of nearly x4 (3.9) compression was achieved with RHIC/STAR data where ROOT compression managed only x1.4.In this contribution, we will present our concepts of “functionally lossless compression”, have a glance at examples and achievements in other communities, present the results and outcome of our current, ongoing R&D, as well as present a high-level view of our plan to move forward with a ROOT implementation that would deliver a basic solution readily integrated into HENP applications. As a collaboration of experimental scientists, private industry, and the ROOT Team, our aim is to capitalize on the substantial success delivered by the initial effort and produce a robust technology properly packaged as an open-source tool that could be used by virtually every experiment around the world as means for improving data management and accessibility.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
NASA NTRS1992

The long hold: Storing data at the National Archives

The National Archives is, in many respects, in a unique position. For example, I find people from other organizations describing an archival medium as one which will last for three to five years. At the National Archives, we deal with the centuries, not years. From our perspective, there is no archival medium for data storage, and we do not expect there will ever be one. Predicting the long-term future of information technology beyond a mere five or ten years approaches the occult arts. But one prediction is probably safe. It is that the technology will continue to change, at least until analysts start talking about the post-information age. If we did have a medium which lasted a hundred years or longer, we probably would not have a device capable of reading it. The issue of obsolescence, as opposed to media stability, is more complex and more costly. It is especially complex at the National Archives because of two other aspects of our peculiar position. The first aspect is that we deal with incoherent data. The second is that we are charged with satisfying unknown and unknowable requirements. A brief overview of these aspects is presented.

Thibodeau, Kenneth↗
NASA NTRS1991

The long hold: Storing data at the National Archives

A description of the information collection and storage needs of the National Archives and Records Administration (NARA) is presented. The unique situation of NARA is detailed. Two aspects which make the issue of obsolescence especially complex and costly are dealing with incoherent data and satisfying unknown and unknowable requirements. The data is incoherent because it comes from a wide range of independent sources, covers unrelated subjects, and is organized and encoded in ways that are not only not controlled but often unknown until received. NARA's mission to preserve and provide access to records with enduring value makes NARA, in effect, the agent of future generations. NARA's responsibility to the future places itself is a perpetual quandary of devotion to serving needs which are unknown.

Thibodeau, Kenneth↗
NASA NTRS2000

Using Hydrogen and Chlorine on Si(111) to Store Data

The interaction of a pyridine molecule with H and Cl atoms on a Si(111) surface is studied using a cluster model in conjunction with the B3LYP/6-3 1G level of theory. Regardless of the type of the neighboring atoms, the interaction of the pyridine with a hydrogen atom is very different from its interaction a chlorine atom. This system is discussed in terms of our atomic data storage proposal.

Bauschlicher, Charles W., Jr.↗