Search NASA⌕ Search

SEARCH · Search NASA

Results for “datastore”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

6 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

The Design and Implementation of a Secure Datastore Based on Ethereum Smart Contract

In this paper, we present a secure datastore based on an Ethereum smart contract. Our research is guided by three research questions. First, we will explore to what extend a smart-contract-based datastore should resemble a traditional database system. Second, we will investigate how to store the data in a smart-contract-based datastore for maximum flexibility while minimizing the gas consumption. Third, we seek answers regarding whether or not a smart-contract-based datastore should incorporate complex processing such as data encryption and data analytic algorithms. The proposed smart-contract-based datastore aims to strike a good balance between several constraints: (1) smart contracts are publicly visible, which may create a confidentiality concern for the data stored in the datastore; (2) unlike traditional database systems, the Ethereum smart contract programming language (i.e., Solidity) offers very limited data structures for data management; (3) all operations that mutate the blockchain state would incur financial costs and the developers for smart contracts must make sure sufficient gas is provisioned for every smart contract call, and ideally, the gas consumption should be minimized. Our investigation shows that although it is essential for a smart-contract-based datastore to offer some basic data query functionality, it is impractical to offer query flexibility that resembles that of a traditional database system. Furthermore, we propose that data should be structured as tag-value pairs, where the tag serves as a non-unique key that describes the nature of the value. We also conclude that complex processing should not be allowed in the smart contract due to the financial burden and security concerns. The tag-based secure datastore designed this way also defines its applicative perimeter, i.e., only applications that align with our strategy would find the proposed datastore a good fit. Those that would rather incur higher financial cost for more data query flexibility and/or less user burden on data pre- and post-processing would find the proposed database too restrictive.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2021

Enabling Scalable and Extensible Memory-mapped Datastores in Userspace

Exascale workloads are expected to incorporate data-intensive processing in close coordination with traditional physics simulations. These emerging scientific, data-analytics and machine learning applications need to access a wide variety of datastores in flat files and structured databases. Programmer productivity is greatly enhanced by mapping datastores into the application process's virtual memory space to provide a unified “in-memory” interface. Currently, memory mapping is provided by system software primarily designed for generality and reliability. However, scalability at high concurrency is a formidable challenge on exascale systems. Also, there is a need for extensibility to support new datastores potentially requiring HPC data transfer services. In this article, we present UMap , a scalable and extensible userspace service for memory-mapping datastores. Furthermore, through decoupled queue management, concurrency aware adaptation, and dynamic load balancing, UMap enables application performance to scale even at high concurrency. We evaluate UMap in data-intensive applications, including sorting, graph traversal, database operations, and metagenomic analytics. Our results show that UMap as a userspace service outperforms an optimized kernel-based service across a wide range of intra-node concurrency by 1.22-1.9 × . We performed two case studies to demonstrate UMap 's extensibility. First, a new datastore residing in remote memory is incorporated into UMap as an application-specific plugin. Second, we present a persistent memory allocator Metall built atop UMap for unified storage/memory.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Optimizing Management of Persistent Data Structures in High-Performance Analytics

Large-scale data analytics workflows ingest massive input data into various data structures, including graphs and key-value datastores. These data structures undergo multiple transformations and computations and are typically reused in incremental and iterative analytics workflows. Persisting in-memory views of these data structures enables reusing them beyond the scope of a single program run while avoiding repetitive raw data ingestion overheads. Memory-mapped I/O enables persisting in-memory data structures without data serialization and deserialization overheads. However, memory-mapped I/O lacks the key feature of persisting consistent snapshots of these data structures for incremental ingestion and processing. The obstacles to efficient virtual memory snapshots using memory-mapped I/O include background writebacks outside the application’s control, and the significantly high storage footprint of such snapshots. To address these limitations, we present Privateer, a memory and storage management tool that enables storage-efficient virtual memory snapshotting while also optimizing snapshot I/O performance. Here, we integrated Privateer into Metall, a state-of-the-art persistent memory allocator for C++, and the Lightning Memory-Mapped Database (LMDB), a widely-used key-value datastore in data analytics and machine learning. Privateer optimized application performance by 1.22× when storing data structure snapshots to node-local storage, and up to 16.7× when storing snapshots to a parallel file system. Privateer also optimizes storage efficiency of incremental data structure snapshots by up to 11× using data deduplication and compression.

Computer science↗
DOE OSTI2020

ServiceX A Distributed, Caching, Columnar Data Delivery Service

We will describe a component of the Intelligent Data Delivery Service being developed in collaboration with IRIS-HEP and the LHC experiments. ServiceX is an experiment-agnostic service to enable on-demand data delivery specifically tailored for nearly-interactive vectorized analysis. This work is motivated by the data engineering challenges posed by HL-LHC data volumes and the increasing popularity of python and Spark-based analysis workflows.ServiceX gives analyzers the ability to query events by dataset metadata. It uses containerized transformations to extract just the data required for the analysis. This operation is colocated with the data to avoid transferring unnecessary branches over the WAN. Simple filtering operations are supported to further reduce the amount of data transferred.Transformed events are cached in a columnar datastore to accelerate delivery of subsequent similar requests. ServiceX will learn commonly related columns and automatically include them in the transformation to increase the potential for cache hits by other users.Selected events are streamed to the analysis system using an efficient wire protocol that can be readily consumed by a variety of computational frameworks. This reduces time-to-insight for physics analysis by delegating to ServiceX the complexity of event selection, slimming, reformatting, and streaming.

Galewsky, B.↗
DOE OSTI2025

GeoGridFusion (Open-Source Geospatial Toolkit for Solar Data Integration​) [SWR-25-19]

GeoGridFusion facilitates the usage and storage of gridded geospatial satellite data by users outside of the National Renewable Lab (NREL), particularly those without access to high-performance computing (HPC) resources. This tool builds on work done by the PVDegradationTools project for DuraMAT, with the goal of making our advancements from this project widely accessible. This repo contains utilities to allow for the storage of user downloaded geospatial weather data by providing a local datastore for storage and spatial queries, supporting large-scale analyses without the need for HPC resources.

Ford, Tobin [National Renewable Energy Laboratory ↗
DOE OSTI2021

Universal Utility Data Exchange (UUDEX) – Security and Administration: Cybersecurity of Energy Delivery Systems (CEDS) Research and Development

A critical component of the Universal Utility Data Exchange (UUDEX) approach is the integrated security contained within its processing. This document describes how that security is designed and expected to be implemented by UUDEX Implementations (U-Implementations), including the UUDEX Server (U-Server) and UUDEX Clients (U-Clients). The UUDEX security hierarchy consists of three levels: 1. The UUDEX Instance (U-Instance) itself, which sits at the top of the hierarchy and contains the U-Server, the UUDEX Identity Authority (U-Identity Authority), and the UUDEX Administrator (U-Administrator) functions; 2. A group of one or more UUDEX Participants (U-Participants) that present “organizations” that participate in the U-Instance and contains the UUDEX Administrator Participant (U-U-Administrator Participant) function; 3. A group of one or more UUDEX Endpoints (U-Endpoints) that represent the individual UUDEX Publish Clients (U-Publish Client) responsible for supplying data to the U-Instance that is consumed by UUDEX Subscriber Clients (U-Subscriber Clients). U-Endpoints can be either autonomous devices that publish and subscribe data such as data exchange servers found in supervisory control and data acquisition and energy management systems, or they can be tied to users of applications that, for example, submit DOE OE-417 disturbance reports. U-Participants and U-Endpoints can be organized into UUDEX Groups (U-Groups). Any number of U-Participants or U-Endpoints can be members of a U-Group. A given U-Participant or U-Endpoint can be a member of multiple U-Groups, but a U-Group cannot contain other U-Groups. For example, a U-Group could be created to contain all U-Participant Transmission Operators within the purview of a Reliability Coordinator, and another U-Group could be created to contain all U-Participant Generator Operators within the purview of a Reliability Coordinator. U-Participants that are both Transmission Operators and Generator Operators would be members of both U-Groups. U-Participants, U-Endpoints, and U-Groups are used in the access control structures to provide access to individual UUDEX Subjects (U-Subjects). U-Groups are created by the U-Administrator and are managed by the U-Administrator or the designated U-Group Managers. U-Endpoints can be assigned UUDEX Roles (U-Roles) that can be used to further restrict access. U-Roles are assigned to individual U-Endpoints. For example, a U-Role of “Security Analyst” could be used to restrict which U-Endpoints can publish or subscribe security incident reports and vulnerability notifications, while a U-Role of “Transmission Planner” can be used to restrict which U-Endpoints can publish power system model updates. U-Role definitions are created by the U-Administrator, but the U-Roles are assigned to U-Endpoints by their respective UUDEX Participant Administrators (U-Participant Administrator). Because all information required to make security decisions is either included within the U-Endpoint’s X.509 digital certificate or stored in a datastore on the U-Server, all security decisions are performed and enforced within the U-Server. This reduces the complexity of the U-Client code and minimizes the chance for compromise of the integrity of the UUDEX security features.

97 MATHEMATICS AND COMPUTING↗