SEARCH · Search NASA

Results for “data storage and retrieval”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

All sources NASA NTRS DOE OSTI

DOE OSTI2020

Systems and methods for data storage and retrieval

A method includes directing a probe beam to a target that includes an array of data portions in a data storage medium arranged so that a beam area of the probe beam extends across a plurality of adjacent data portions, the array including a data portion subset with each data portion of the subset responsive to the probe beam to produce a response illumination, receiving the response illumination at a detector, and determining data values corresponding to the plurality of adjacent data portions based on the received response illumination. Apparatus and systems are also disclosed.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

A Ceph S3 Object Data Store for HEP

We present a novel data format design that obviates the need for data tiers by storing individual event data products in column objects. The objects are stored and retrieved through Ceph S3 technology, with a layout designed to minimize metadata volume and maximize data processing parallelism. Performance benchmarks of data storage and retrieval are presented.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2024

Building a FAIR data ecosystem for incorporating single-cell transcriptomics data into agricultural genome to phenome research

Introduction The agriculture genomics community has numerous data submission standards available, but the standards for describing and storing single-cell (SC, e.g., scRNA- seq) data are comparatively underdeveloped. Methods To bridge this gap, we leveraged recent advancements in human genomics infrastructure, such as the integration of the Human Cell Atlas Data Portal with Terra, a secure, scalable, open-source platform for biomedical researchers to access data, run analysis tools, and collaborate. In parallel, the Single Cell Expression Atlas at EMBL-EBI offers a comprehensive data ingestion portal for high-throughput sequencing datasets, including plants, protists, and animals (including humans). Developing data tools connecting these resources would offer significant advantages to the agricultural genomics community. The FAANG data portal at EMBL-EBI emphasizes delivering rich metadata and highly accurate and reliable annotation of farmed animals but is not computationally linked to either of these resources. Results Herein, we describe a pilot-scale project that determines whether the current FAANG metadata standards for livestock can be used to ingest scRNA-seq datasets into Terra in a manner consistent with HCA Data Portal standards. Importantly, rich scRNA-seq metadata can now be brokered through the FAANG data portal using a semi-automated process, thereby avoiding the need for substantial expert curation. We have further extended the functionality of this tool so that validated and ingested SC files within the HCA Data Portal are transferred to Terra for further analysis. In addition, we verified data ingestion into Terra, hosted on Azure, and demonstrated the use of a workflow to analyze the first ingested porcine scRNA-seq dataset. Additionally, we have also developed prototype tools to visualize the output of scRNA-seq analyses on genome browsers to compare gene expression patterns across tissues and cell populations. This JBrowse tool now features distinct tracks, showcasing PBMC scRNA-seq alongside two bulk RNA-seq experiments. Discussion We intend to further build upon these existing tools to construct a scientist-friendly data resource and analytical ecosystem based on Findable, Accessible, Interoperable, and Reusable (FAIR) SC principles to facilitate SC-level genomic analysis through data ingestion, storage, retrieval, re-use, visualization, and comparative annotation across agricultural species.

Genetics & Heredity↗

DOE OSTI2020

Video Summarization Using Deep Action Recognition Features and Robust Principal Components Analysis

In an instance where desired pre-defined actions, behaviors, or other categories are known a priori, various video classification and recognition models can be trained to discover those classifications and their location within the video. Absent that information, one might still be tasked with identifying interesting portions within a video, a process which—if done manually—is onerous and time-consuming as it requires manual inspection of the video itself. Recognizing high-level interesting segments within a whole video has been a general area of interest due to the ubiquity of video data. However the size of the data makes storage, retrieval, and inspection of large collections of videos cumbersome. This problem motivates the task of generating shortened clips highlighting the primary content of a video, relieving the burden of having to watch the entire video. This paper presents an unsupervised method of creating shortened clips of videos, enabling the rapid review of the most interesting content within a video. Our method uses features extracted from pre-trained action recognition models as input to online moving window robust principal component analysis to generate summaries. The procedure is tested on a publicly available video summarization dataset and demonstrates comparable performance to state-of-the-art in an un-augmented setting while requiring no training.

Claborne, Daniel M.↗

DOE OSTI2020

Java Software Extensibility Library

The Java Software Extensibility Library provides tools that help Java software developers produce flexible and extensible Java applications. The major components include a Plugin Library and a Polymorphic Map data structure. The Plugin Library provides developers with a Plugin Registry and a standard pattern for designing, versioning, registering, retrieving, and annotating plugins. It also enables advanced capabilities such as loading plugins at runtime, and creating/loading encrypted plugin packages. The Polymorphic Map data structure facilitates the storage and retrieval of objects of varying types within a single data structure; while maintaining type safety. The library also provides a standard design pattern for producing application-specific "Domain Maps" and "Closed Domain Maps".SAND2020-3816 M Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Fleschute, Christopher↗

DOE OSTI2021

Securing Environmental IoT Data Using Masked Authentication Messaging Protocol in a DAG-Based Blockchain: IOTA Tangle

The demand for the digital monitoring of environmental ecosystems is high and growing rapidly as a means of protecting the public and managing the environment. However, before data, algorithms, and models can be mobilized at scale, there are considerable concerns associated with privacy and security that can negatively affect the adoption of technology within this domain. In this paper, we propose the advancement of electronic environmental monitoring through the capability provided by the blockchain. The blockchain’s use of a distributed ledger as its underlying infrastructure is an attractive approach to counter these privacy and security issues, although its performance and ability to manage sensor data must be assessed. We focus on a new distributed ledger technology for the IoT, called IOTA, that is based on a directed acyclic graph. IOTA overcomes the current limitations of the blockchain and offers a data communication protocol called masked authenticated messaging for secure data sharing among Internet of Things (IoT) devices. We show how the application layer employing the data communication protocol, MAM, can support the secure transmission, storage, and retrieval of encrypted environmental sensor data by using an immutable distributed ledger such as that shown in IOTA. Finally, we evaluate, compare, and analyze the performance of the MAM protocol against a non-protocol approach.

Gangwani, Pranav (ORCID:0000000159226002)↗

DOE OSTI2020

Prediction of room-temperature multiferroicity in strained MoCr 2 S 6 monolayer

The contrasting d-orbital occupation required for ferroelectricity vs ferromagnetism makes it difficult for their coexistence in two-dimensional materials, especially at high temperature. To resolve this intrinsic contradiction, we propose a layered MoCr 2 S 6 multiferroics by alloying magnetic Cr element into the ferroelectric 1T phase of the MoS 2 matrix. First-principles calculations disclose that a spontaneous symmetry breaking, depending on the Mo atom displacement, leads to a robust ferroelectricity, which coexists with a ferromagnetic order originated from two neighboring Cr atoms. The effect can be further enhanced by tensile strain to bring about a room-temperature multiferroicity. Our results shed new light on the fundamental understanding of multiferroics and display promising applications in spintronics and multistate data storage.

2D materials↗

DOE OSTI2020

Skyrmion fluctuations at a first-order phase transition boundary

Magnetic skyrmions are topologically protected spin textures with promising prospects for applications in data storage. They can form a lattice state due to competing magnetic interactions and are commonly found in a small region of the temperature—magnetic field phase diagram. Recent work has demonstrated that these magnetic quasi-particles fluctuate at the μeV energy scale. Here, we use a coherent x-ray correlation method at an x-ray free-electron laser to investigate these fluctuations in a magnetic phase coexistence region near a first-order transition boundary where fluctuations are not expected to play a major role. Surprisingly, we find that the relaxation of the intermediate scattering function at this transition differs significantly compared to that deep in the skyrmion lattice phase. The observation of a compressed exponential behavior suggests solid-like dynamics, often associated with jamming. We assign this behavior to disorder and the phase coexistence observed in a narrow field-window near the transition, which can cause fluctuations that lead to glassy behavior.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2022

Controlling magnetic configuration in soft–hard bilayers probed by polarized neutron reflectometry

Hard/soft magnetic bilayer thin films have been widely used in data storage technologies and permanent magnet applications. The magnetic configuration and response to temperatures and magnetic fields in these heterostructures are considered to be highly dependent on the interfacial coupling. However, the intrinsic properties of each of the layers, such as the saturation magnetization and layer thickness, also strongly influence the magnetic configuration. Changing these parameters provides an effective method to tailor magnetic properties in composite magnets. Here, we use polarized neutron reflectometry (PNR) to experimentally probe the interfacial magnetic configurations in the hard/soft bilayer thin films: L1 0 -FePt/A1-FePt, [Co/Pd]/CoPd, [Co/Pt]/FeNi, and L1 0 -FePt/Fe, all of which have a perpendicular magnetic anisotropy in the hard layer. These films were designed with different soft and hard layer thicknesses (t soft and t hard ) and saturation magnetization (M$_s^{soft}$ and M$_s^{hard}$). The influences of an in-plane magnetic field (H ip ) and temperature (T) are also studied using a L1 0 -FePt/A1-FePt bilayer sample. Comparing the PNR results to the micromagnetic simulations reveals that the interfacial magnetic configuration is highly dependent on t soft , M$_s^{soft}, and the external factors (H ip and T) and has a relatively weak dependence on t hard and M$_s^{hard}$. Key among these results, for thin t soft , the hard and soft layers are rigidly coupled in the out-of-plane direction and then undergo a transition to relax in-plane. This transition can be delayed to larger t soft by decreasing M$_s^{soft}$. Understanding the influence of these parameters on the magnetic configuration is critical to designing functional composite magnets for applications.

36 MATERIALS SCIENCE↗

DOE OSTI2023

The fast camera (Fastcam) imaging diagnostic systems on the DIII-D tokamak

Two camera systems are installed on the DIII-D tokamak at the toroidal positions of 90° (90° system) and 225° (225° system), respectively. The cameras have two types of relay optics, namely, a coherent optical fiber bundle and a periscope system. The periscope system provides absolute intensity calibration stability while sacrificing resolution (10 lp/mm), while the fiber system provides high resolution (16 lp/mm) while sacrificing calibration stability. The periscope is available only for the 90° system. The optics of the 225° system were designed for view stability, repeatability, and easy maintenance. The cameras are located inside optimized neutron, x ray and magnetic shielding in order to reduce electronics damage, reboots, and magnetic and neutron interference, increasing the overall system reliability. An automated filter wheel, providing remote filter change, allows for remote wavelength selection. A software suite automates camera acquisition and data storage, allowing for remote operation and reduced operator involvement. System metadata is used to streamline the data analysis workflow, particularly for intensity calibration. Here, the spatial calibration uses multiple observable wall features, resulting in a reconstruction accuracy ≤2 cm.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2024

The role of magnetic anisotropy in the magnetoresistance of Cr 2 O 3 /Al 2 O 3 thin film antiferromagnets

The magnetic states of antiferromagnetic insulating thin films are a promising medium for information storage, but characterization of these states has proven to be challenging. One approach is via magnetotransport measurements in an adjacent heavy metal layer. To this end, we synthesized and characterized a series of Cr 2 O 3 films and bilayers on Al 2 O 3 substrates with three different orientations: m-plane, a-plane, and c-plane. X-ray diffraction results demonstrated orientation control of the Cr 2 O 3 thin film, with m-plane films displaying a higher degree of mosaic spread than the a- and c-plane films. Reciprocal space maps showed that the films are mostly relaxed, although there was a small and different degree of strain in each orientation. The m-plane films were under 2% compressive strain, the a-plane film was under 0.5% compressive strain, and the c-plane film was completely relaxed to bulk values. To probe the magnetic state of the films, we measured the angular dependent magnetoresistance of Cr 2 O 3 /Pt bilayers for each orientation. We found a nontrivial temperature dependence of the sign of the magnetoresistance, pointing to the complex interplay between the exchange and anisotropy energies that vary with orientation. We propose that strain and mosaic spread may contribute to a difference in magnetic anisotropies among the samples and the resulting temperature dependence of the magnetoresistance. This work demonstrates the importance of considering the competition between antiferromagnetic exchange and magnetic anisotropy when storing information in the spin state of an antiferromagnetic insulator.

42 ENGINEERING↗

DOE OSTI2024

Effects of carbon concentration on the local atomic structure of amorphous GST

Ge-Sb-Te (GST) alloys are leading phase-change materials for data storage due to the fast phase transition between amorphous and crystalline states. Ongoing research aims at improving the stability of the amorphous phase to improve retention. This can be accomplished by the introduction of carbon as a dopant to Ge 2 Sb 2 Te 5 , which is known to alter the short- and mid-range structure of the amorphous phase and form covalently bonded C clusters, both of which hinder crystallization. The relative importance of these processes as a function of C concentration is not known. Here we used molecular dynamics simulation based on density functional theory to study how carbon doping affects the atomic structure of GST-C. Carbon doping results in an increase in tetrahedral coordination, especially of Ge atoms, and this is known to stabilize the amorphous phase. We observe an unexpected, non-monotonous trend in the number of tetrahedral bonded Ge with the amount of carbon doping. Our simulations show an increase in the number of tetrahedral bonded Ge up to 5 at.% C, after which the number saturates and begins to decrease above 14 at.% C. The carbon atoms aggregate into clusters, mostly in the form of chains and graphene flakes, leaving less carbon to disrupt the GST matrix at higher carbon concentrations. Different degrees of carbon clustering can explain divergent experimental results for recrystallization temperature for carbon doped GST.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2020

Compressing data for storage in cache memories in a hierarchy of cache memories

An electronic device includes at least one compression-decompression functional block and a hierarchy of cache memories with a first cache memory and a second cache memory. The at least one compression-decompression functional block receives data in an uncompressed state, compresses the data using one of a first compression or a second compression, and, after compressing the data, provides the data to the first cache memory for storage therein. When the data is retrieved from the first cache memory to be stored in the second cache memory, when the data is compressed using the first compression, the compression-decompression functional block decompresses the data to reverse effects of the first compression on the data, thereby restoring the data to the uncompressed state and provides the data compressed using the second compression or in the uncompressed state to the second cache memory for storage therein.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Viability of S3 Object Storage for the ASC Program at Sandia

Recent efforts at Sandia such as DataSEA are creating search engines that enable analysts to query the institution’s massive archive of simulation and experiment data. The benefit of this work is that analysts will be able to retrieve all historical information about a system component that the institution has amassed over the years and make better-informed decisions in current work. As DataSEA gains momentum, it faces multiple technical challenges relating to capacity storage. From a raw capacity perspective, data producers will rapidly overwhelm the system with massive amounts of data. From an accessibility perspective, analysts will expect to be able to retrieve any portion of the bulk data, from any system on the enterprise network. Sandia’s Institutional Computing is mitigating storage problems at the enterprise level by procuring new capacity storage systems that can be accessed from anywhere on the enterprise network. These systems use the simple storage service, or S3, API for data transfers. While S3 uses objects instead of files, users can access it from their desktops or Sandia’s high-performance computing (HPC) platforms. S3 is particularly well suited for bulk storage in DataSEA, as datasets can be decomposed into object that can be referenced and retrieved individually, as needed by an analyst. In this report we describe our experiences working with S3 storage and provide information about how developers can leverage Sandia’s current systems. We present performance results from two sets of experiments. First, we measure S3 throughput when exchanging data between four different HPC platforms and two different enterprise S3 storage systems on the Sandia Restricted Network (SRN). Second, we measure the performance of S3 when communicating with a custom-built Ceph storage system that was constructed from HPC components. Overall, while S3 storage is significantly slower than traditional HPC storage, it provides significant accessibility benefits that will be valuable for archiving and exploiting historical data. There are multiple opportunities that arise from this work, including enhancing DataSEA to leverage S3 for bulk storage and adding native S3 support to Sandia’s IOSS library.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2020

Science-Driven Data Management for Multi-Tiered Storage (Final Report)

Scientific discovery at the exascale will not be possible without significant new research in the management, storage and retrieval over the long lifespan of the extreme amounts of data that will be produced. Our thesis is that adding application level knowledge about data to guide the actions of the storage system provides substantial benefits to the organization, storage, and access to extreme scale data, resulting in improved productivity for computational science. In this project we will demonstrate novel techniques to facilitate efficient mapping of data objects, even partitioning individual variables, from the user space onto multiple storage tiers, and enable application-guided data reductions and transformations to address capacity and bandwidth bottlenecks. Our goal is to address the associated Input/ Output (I/O) and storage challenges in the context of current and emerging storage landscapes, and expedite insights into mission critical scientific processes.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2021

Error-controlled, progressive, and adaptable retrieval of scientific data with multilevel decomposition

Extreme-scale simulations and high-resolution instruments have been generating an increasing amount of data, which poses significant challenges to not only data storage during the run, but also post-processing where data will be repeatedly retrieved and analyzed for a long period of time. The challenges in satisfying a wide range of post-hoc analysis needs while minimizing the I/O overhead caused by inappropriate and/or excessive data retrieval should never be left unmanaged. In this paper, we propose a data refactoring, compressing, and retrieval framework capable of 1) fine-grained data refactoring with regard to precision; 2) incrementally retrieving and recomposing the data in terms of various error bounds; and 3) adaptively retrieving data in multi-precision and multi-resolution with respect to different analysis. With the progressive data re-composition and the adaptable retrieval algorithms, our framework significantly reduces the amount of data retrieved when multiple incremental precision are requested and/or the downstream analysis time when coarse resolution is used. Experiments show that the amount of data retrieved under the same progressively requested error bound using our framework is 64% less than that using state-of-the-art single-error-bounded approaches. Parallel experiments with up to 1, 024 cores and ~ 600 GB data in total show that our approach yields 1.36× and 2.52× performance over existing approaches in writing to and reading from persistent storage systems, respectively.

Liang, Xin↗

DOE OSTI2023

Improving Progressive Retrieval for HPC Scientific Data using Deep Neural Network

As the disparity between compute and I/O on high-performance computing systems has continued to widen, it has become increasingly difficult to perform post-hoc data analytics on full-resolution scientific simulation data due to the high I/O cost. Error-bounded data decomposition and progressive data retrieval framework has recently been developed to address such a challenge by performing data decomposition before storage and reading only part of the decomposed data when necessary. However, the performance of the progressive retrieval framework has been suffering from the over-pessimistic error control theory, such that the achieved maximum error of recomposed data is significantly lower than the required error. Therefore, more data than required is fetched for recomposition, incurring additional I/O overhead. In order to tackle this issue, we propose a DNN-based progressive retrieval framework that can better identify the minimum amount of data to be retrieved. Our contributions are as follows: 1) We provide an in-depth investigation of the recently developed progressive retrieval framework; 2) We propose two designs of prediction models (named D-MGARD and E-MGARD) to estimate the amount of retrieved data size based on error bounds. 3) We evaluate our proposed solutions using scientific datasets generated by real-world simulations from two domains. Evaluation results demonstrate the effectiveness of our solution in accurately predicting the amount of retrieval data size, as well as the advantages of our solution over the traditional approach to reducing the I/O overhead. Based on our evaluation, our solution is shown to read significantly less data (5% - 40% with D-MGARD, 20% - 80% with E-MGARD).

Wang, Jinzhen↗

DOE OSTI2024

Error-controlled Progressive Retrieval of Scientific Data under Derivable Quantities of Interest

The unprecedented amount of scientific data has introduced heavy pressure on the current data storage and transmission systems. Progressive compression has been proposed to mitigate this problem, which offers data access with on-demand precision. However, existing approaches only consider precision control on primary data, leaving uncertainties on the quantities of interest (QoIs) derived from it. In this work, we present a progressive data retrieval framework with guaranteed error control on derivable QoIs. Our contributions are three-fold. (1) We carefully derive the theories to strictly control QoI errors during progressive retrieval. Our theory is generic and can be applied to any QoIs that can be composited by the basis of derivable QoIs proved in the paper. (2) We design and develop a generic progressive retrieval framework based on the proposed theories, and optimize it by exploring feasible progressive representations. (3) We evaluate our framework using five real-world datasets with a diverse set of QoIs. Experiments demonstrate that our framework can faithfully respect any user-specified QoI error bounds in the evaluated applications. This leads to over 2.02× performance gain in data transfer tasks compared to transferring the primary data while guaranteeing a QoI error that is less than 1E-5.

Wu, Xuan↗