Search NASA⌕ Search

SEARCH · Search NASA

Results for “data storage data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

mzPeak: Designing a Scalable, Interoperable, and Future-Ready Mass Spectrometry Data Format

Advances in mass spectrometry (MS) instrumentation, such as higher resolution, faster scan speeds, and improved sensitivity, have significantly increased the volume and complexity of data. The growing adoption of imaging and ion mobility further amplifies these challenges across MS-based omics fields, including proteomics, metabolomics, and lipidomics. While these technologies unlock new possibilities, they also present significant challenges in data management, storage, and accessibility. Existing open formats, such as the XML-based community standards mzML and imzML, struggle to meet the demands of modern MS workflows due to their large file sizes, slow data access, and limited metadata support. Vendor-specific formats, while optimized for proprietary instruments, lack interoperability, comprehensive metadata support and long-term archival reliability. This white paper lays the groundwork for mzPeak, a next-generation community data format designed to address these challenges and support high-throughput, multi-dimensional MS workflows. By adopting a hybrid model that combines efficient binary storage for numerical data and both human and machine-readable metadata storage, mzPeak will reduce file sizes, accelerate data access, and offer a scalable, adaptable solution for evolving MS technologies. For researchers, mzPeak will enable enhanced interoperability across platforms, seamless support for complex workflows including ion mobility and MS imaging, and faster data access compared to existing community formats such as mzML. Its design will ensure data is managed in compliance with regulatory standards, essential for applications such as precision medicine and chemical safety, where long-term data integrity and accessibility are critical. For vendors, mzPeak provides a streamlined, open alternative to proprietary formats, reducing the burden of regulatory compliance while aligning with the industry's push for transparency and standardization. By offering a high-performance, interoperable solution, mzPeak positions vendors to meet customer demands for sustainable data management tools which will be able to handle emerging and future data types and workflows. mzPeak aspires to become the cornerstone of MS data management, empowering researchers, vendors, and developers to innovate and collaborate more effectively.

data formats↗

Orchestration of materials science workflows for heterogeneous resources at large scale

In the era of big data, materials science workflows need to handle large-scale data distribution, storage, and computation. Any of these areas can become a performance bottleneck. We present a framework for analyzing internal material structures (e.g., cracks) to mitigate these bottlenecks. We demonstrate the effectiveness of our framework for a workflow performing synchrotron X-ray computed tomography reconstruction and segmentation of a silica-based structure. Our framework provides a cloud-based, cutting-edge solution to challenges such as growing intermediate and output data and heavy resource demands during image reconstruction and segmentation. Specifically, our framework efficiently manages data storage, scaling up compute resources on the cloud. The multi-layer software structure of our framework includes three layers. A top layer uses Jupyter notebooks and serves as the user interface. A middle layer uses Ansible for resource deployment and managing the execution environment. A low layer is dedicated to resource management and provides resource management and job scheduling on heterogeneous nodes (i.e., GPU and CPU). At the core of this layer, Kubernetes supports resource management, and Dask enables large-scale job scheduling for heterogeneous resources. The broader impact of our work is four-fold: through our framework, we hide the complexity of the cloud’s software stack to the user who otherwise is required to have expertise in cloud technologies; we manage job scheduling efficiently and in a scalable manner; we enable resource elasticity and workflow orchestration at a large scale; and we facilitate moving the study of nonporous structures, which has wide applications in engineering and scientific fields, to the cloud. While we demonstrate the capability of our framework for a specific materials science application, it can be adapted for other applications and domains because of its modular, multi-layer architecture.

97 MATHEMATICS AND COMPUTING↗

BisQue for 3D Materials Science in the Cloud: Microstructure–Property Linkages

Accelerating the design and development of new advanced materials is one of the priorities in modern materials science. These efforts are critically dependent on the development of comprehensive materials cyber-infrastructures which enable effcient data storage, management, sharing, and collaboration as well as integration of computational tools that help establish processing{structure{property relationships. In this contribution, we present implementation of such computational tools into a cloud-based platform called BisQue (Kvilekval et al., 2010). We first describe the current state of BisQue as an open-source platform for multidisciplinary research in the cloud and its potential for 3D materials science. We then demonstrate how new computational tools, primarily aimed at processing{structure{property relationships, can be implemented into the system. Speci cally, in this work, we develop a module for BisQue that enables microstructure-sensitive predictions of effective yield strength of two-phase materials. Towards this end, we present an implementation of a computationally effcient data-driven model into the BisQue platform. The new module is made available online and can be used from a web browser without any special software and with minimal computational requirements on the user end. The capabilities of the module for rapid property screening are demonstrated in case studies with two different methodologies based on datasets containing 3D microstructure information from (i) synthetic generation and (ii) sampling large 3D volumes obtained in experiments.

36 MATERIALS SCIENCE↗

Materials Databases Infrastructure Constructed by First Principles Calculations: A Review

The First Principles calculations, especially the calculation based on High-Throughput Density Functional Theory, have been widely accepted as the major tools in atom scale materials design. The emerging super computers, along with the powerful First Principles calculations, have accumulated hundreds of thousands of crystal and compound records. The exponential growing of computational materials information urges the development of the materials databases, which not only provide unlimited storage for the daily increasing data, but still keep the efficiency in data storage, management, query, presentation and manipulation. This review covers the most cutting edge materials databases in materials design, and their hot applications such as in fuel cells. By comparing the advantages and drawbacks of these high-throughput First Principles materials databases, the optimized computational framework can be identified to fit the needs of fuel cell applications. The further development of high-throughput DFT materials database, which in essence accelerates the materials innovation, is discussed in the summary as well.

36 MATERIALS SCIENCE↗

Management and Storage of Scientific Data

Scientific discoveries rely heavily on efficient access, search, and management of massive data sets. Data management technologies have, for decades, provided foundational capabilities for scientific computing. Just as storage, input/output (I/O), and data management have been fundamental to simulation-based science for many years, so too are capable data-management technologies key to the success of today’s scientific workflows utilizing data intensive and machine learning (ML) techniques. The Department of Energy, Office of Science, Advanced Scientific Computing Research (ASCR) program has invested broadly in data-management research focused on high-performance computing (HPC) systems, from parallel file systems that store data to application software that makes these systems more productive. Still, advances in technology combined with growing diversity of supported science strongly motivate continued investment in this area. In January 2022, ASCR convened a workshop to identify priority research directions in the area of data management for high-performance and scientific computing. Attendees were challenged to identify promising approaches that would support the breadth of the DOE mission, including the explosion of artificial intelligence (AI) uses and the growing needs of experimental and observational science. Technological and science drivers were identified and considered as they relate to key aspects of data management such as interfaces, architectural design, and FAIR principles (Findable, Accessible, Interoperable, and Reusable). The thoughts of the workshop participants were distilled into a set of four priority research directions with the potential for high impact on DOE science. These research directions are summarized in the following pages.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Programming Abstractions for Managing Workflows on Tiered Storage Systems

Scientific workflows in High Performance Computing (HPC) environments are processing large amounts of data. The storage hierarchy on HPC systems is getting deeper, driven by new technologies (NVRAMs, SSDs, etc.) There is a need for new programming abstractions that allow users to seamlessly manage data at the workflow level on multi-tiered storage systems, and provide optimal workflow performance and use of storage resources. In previous work, we introduced a software architecture Managing Data on Tiered Storage for Scientific Workflows (MaDaTS) that used a Virtual Data Space (VDS) abstraction to hide the complexities of the underlying storage system while allowing users to control data management strategies. In this article, we detail the data-centric programming abstractions that allow users to manage a workflow around its data on the storage layer. The programming abstractions simplify data management for scientific workflows on multi-tiered storage systems, without affecting workflow performance or storage capacity. We measure the overheads and effectiveness introduced by the programming abstractions of MaDaTS. Our results show that these abstractions can optimally use the storage capacity in lesser capacity storage tiers, and simplify data management without adding any performance overheads.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

ATLAS Data Carousel

The ATLAS experiment at CERN's LHC stores detector and simulation data in raw and derived data formats across more than 150 Grid sites world-wide, currently in total about 200PB on disk and 250PB on tape. Data have different access characteristics due to various computational workflows, and can be accessed from different media, such as remote I/O, disk cache on hard disk drives or SSDs. Also, larger data centers provide the majority of offline storage capability via tape systems. For the High-Luminosity LHC (HL-LHC), the estimated data storage requirements are several factors bigger than the present forecast of available resources, based on a flat budget assumption. On the computing side, ATLAS Distributed Computing was very successful in the last years with high performance and high throughput computing integration and in using opportunistic computing resources for the Monte Carlo simulation. On the other hand, equivalent opportunistic storage does not exist. ATLAS started the "Data Carousel" project to increase the usage of less expensive storage, i.e. tapes or even commercial storage, so it is not limited to tape technologies exclusively. Data Carousel orchestrates data processing between workload management, data management, and storage services with the bulk data resident on offline storage. The processing is executed by staging and promptly processing a sliding window of inputs onto faster buffer storage, such that only a small percentage of input data are available at any one time. With this project, we aim to demonstrate that this is the natural way to dramatically reduce our storage cost. The first phase of the project was started in the fall of 2018 and was related to I/O tests of the sites archiving systems. Phase II now requires a tight integration of the workload and data management systems. Additionally, the Data Carousel studies the feasibility to run multiple computing workflows from tape. The project is progressing very well and the results presented in this document will be used before the LHC Run 3.

97 MATHEMATICS AND COMPUTING↗

Livewire Data Platform-A Solution for Energy Efficient Mobility Systems (EEMS) Data Sharing

This presentation is an overview of progress made on the Livewire Data Platform since June 2021. Livewire is a publicly accessible platform for sharing energy efficiency and mobility research data funded by DOE's Vehicle Technologies Office. Core services and platform capabilities of Livewire include: Free, secure data storage; Access management that allows data owners to control who sees their data; Data collection and preservation; Quality characterization; Detailed access and download metrics; and Increased visibility of projects and data.

energy efficiency mobility↗

Livewire Data Platform-A Solution for Energy Efficient Mobility Systems (EEMS) Data Sharing

This presentation is an overview of progress made on the Livewire Data Platform since June 2022. Livewire is a publicly accessible platform for sharing energy efficiency and mobility research data funded by DOE's Vehicle Technologies Office. Core services and platform capabilities of Livewire include: Free, secure data storage; Access management that allows data owners to control who sees their data; Data collection and preservation; Quality characterization; Detailed access and download metrics; and Increased visibility of projects and data. Anyone can create an account and access data at https://livewire.energy.gov/.

energy efficiency↗

ARCHIE: Data Analysis Acceleration with Array Caching in Hierarchical Storage

© 2018 IEEE. Scientific data analysis typically involves reading massive amounts of data that was generated by simulations, experiments, and observations. Performance of reading such large volumes of data from disk-based file systems is often poor because of the slow and mechanical components in the disks. Recent supercomputing systems are adding non-volatile storage layers in a hierarchy to handle the performance gap between fast main memory and slow disk-based storage. Software libraries for managing this hierarchy not only need efficient reading of data but also reduce user-involvement for cross-layer data movement. Furthermore, these libraries need to support array data access patterns into hierarchical storage management as scientific data is often organized in array-based data structures. Existing software typically manage individual storage layers requiring significant manual process in moving data among them. In this paper, we introduce a new array caching in hierarchical storage (ARCHIE) to accelerate array data analysis in a seamless fashion. ARCHIE evaluates array access patterns and prefetches data with array semantics between storage layers. Our evaluation shows that ARCHIE outperforms state-of-the-art file systems, i.e., Lustre and DataWarp, on a production supercomputing system by up to 5.8× in accessing data by scientific analysis applications.

Dong, Bin↗

Development of a Technology Roadmap for Online Monitoring of Nuclear Power Plants

The nuclear power industry has recently recognized the potential cost saving that can be achieved from migrating the current manual and labor dependent surveillance and preventive maintenance activities to data-driven online monitoring methods. Since then, several efforts have been launched with various degrees of momentum to tackle specific surveillance or preventive maintenance activities. The light water reactor sustainability program (LWRS) program has anticipated this need and launched an effort that aimed to develop a technology roadmap for the nuclear power industry. Because the roadmap has to be focused towards the user or application’s value, and because nuclear power plants have different process efficiencies and deficiencies, the technology roadmap has to be plant independent. Achieving this objective required optimizing the extent of details described by a technology roadmap. The approach followed is for the roadmap to describe the sequence of activities needed to automate a process or equipment activity, but not describe the sequence of processes or equipment to target. For example, the steps required to migrate the inspection of a pump are plant independent. However, defining that the migration of feed water pumps inspections to online monitoring should be performed before inspections of feed water valves is dependent on multiple plant factors such as the plant requirements, labor work capacity, and process or equipment conditions. The technology roadmap was broken down into six elements, four of which have a sequential path of advancement, while the other two are supporting elements to the sequential elements. The sequential part of the roadmap analyzes data collection, analytics, visualization, and management. For these elements of the technology roadmap, the migration of an activity is defined in three states. The base state is defined as the most primitive, manual, and labor dependent process used by some nuclear power plants. The modern state is defined as the state that can be achieved if the activities are augmented or replaced by current technologies that are commercially available or are soon to be available. The state of the art is defined as using technologies of the future, i.e. concept and technologies that are being researched. The data collection part of the technology roadmap defines three states of every process performed in a nuclear power plant. However, the data analytics part is described based on equipment, because automating an activity performed on one equipment required considering multiple data sources. The visualization and data management elements are process or equipment independent. Visualization describes the means to use human factor science and technologies advancement in presenting the data collected and the data analytics results. The data management targets storage, communication, and computational power need, in addition to data integration and sharing. Since storage, communication, and computational power technologies and path for deployment are known to any information technology organization, this effort targeted the cost of data management. This has a direct impact on the feasibility of the automation effort. The supporting elements analyzes the value of automating the activity and the change challenges to tackle. For the support elements, the roadmap targeted describing activities that need to be performed as part of the roadmap development. The value element of the roadmap describes how to analyze the cost saving and the associated risk. The change management identifies multiple factors to consider such as regulation, cybersecurity, resource availability, supporting infrastructure availability, feasibility (power, fitness, environment tolerance, facilities), and cultural change.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Opportunities and data requirements for data-driven prognostics and health management in liquid hydrogen storage systems

During the past decade, Prognostics and Health Management (PHM) has become an important set of tools in various areas of industry and academic reliability engineering. PHM consists of a variety of mathematical and computational methods used to support data-driven decision-making to increase the safety, availability, and reliability of complex engineering systems. In particular, PHM can provide crucial insight into reliability and safety design improvements for developing technologies where historical performance and failure data are limited. This is the case of hydrogen fueling and storage technologies. This work presents a high-level approach for designing data-driven PHM applications for bulk liquid hydrogen (LH 2 ) storage systems for hydrogen fueling stations. This paper addresses core aspects of the design, development, and implementation of data-driven PHM applications that can improve the reliability assessment of hydrogen components. The analysis focuses on the relationship between data availability and diagnostic/prognostic capabilities; potential challenges; and integration schemes for current risk mitigation measures. We identify potential condition-monitoring data sources for key components in an LH 2 storage system, including storage tanks, piping, and pumps. Further, we determine that the short-term goals for the implementation of data-driven models in PHM frameworks in hydrogen systems should focus on developing adequate data collection and analysis strategies, as well as exploring the effect on reliability, safety, and regulations for hydrogen systems.

08 HYDROGEN↗

Southeast Regional CO 2 Utilization and Storage Acceleration Partnership (SECARB-USA): Data Management Plan

The “Southeast Regional CO 2 Utilization and Storage Acceleration Partnership” (SECARB-USA) is designed to assess the region’s subsurface resources and burgeoning technology applications. This work supports the U.S. Department of Energy’s (DOE) Office of Fossil Energy’s (FE) mission to help the United States meet its need for secure, affordable, and environmentally sound fossil energy supplies by utilizing the advancements made by the current RCSP initiative to continue to identify and address knowledge gaps.

42 ENGINEERING↗

Integrating Data Management and Collaborative Sharing with Computational Science Processes

Structured Scientific Data Management - the management of storage, access, usage, lifecycle, content and meaning for scientific data - is not as commonly employed in computational science as it is in other fields of scientific endeavor. However, where it has been co-developed and integrated with the computational science research it has had a transformational influence on the scientific work. These infrastructures enabled not only new research previously impossible, but also helped to speed up the research process and improved the quality of the research output. Good data management systems are capable of facilitating effective scientific collaborations on a group, institutional, national or international level, through the easy sharing of resources and results. Today as computational science is becoming more data rich and collaborative, integrated scientific data management is becoming an essential tool for every computational science research and production environment. This chapter will describe the fundamental principles and components of a good data management system, provide real world examples of successful implementations and provides an outlook on future developments. We conclude with a short section on how to get started for those whose interest has been peaked by this chapter

42 ENGINEERING↗

National Renewable Energy Laboratory, South Table Mountain Campus, Golden, Colorado

The National Renewable Energy Laboratory (NREL) South Table Mountain campus, located in Golden, Colorado, has multiyear datasets from buildings with approximately 1,100,000 sq ft of floor area. These 16 buildings include a large office, nine laboratories, and other education or public assembly facilities such as an education center, a warehouse, quick service restaurants, two site entrance security buildings, and a parking garage—all constructed and improved from 1982 to 2014. These datasets are also available from the NREL data storage and management platform SkySparks. The datasets contain electricity meter data (poser, voltage, and current) with 1-minute resolution and building automation system (BAS) data with resolutions of 5 or 15 minutes. Some subsystems (e.g., lighting) or equipment electricity data have the same intervals as above. Some mechanical equipment state or condition data, such as flow and temperature, are captured with 5- to 15-minute intervals. The types of measurement points (i.e., electrical meter power and zone temperature) and associated data quality across the campus are typical, which helps reduce the unreliability of data quality.

STM, Systems and equipment operational, Energy use↗

Science-Driven Data Management for Multi-Tiered Storage (Final Report)

Scientific discovery at the exascale will not be possible without significant new research in the management, storage and retrieval over the long lifespan of the extreme amounts of data that will be produced. Our thesis is that adding application level knowledge about data to guide the actions of the storage system provides substantial benefits to the organization, storage, and access to extreme scale data, resulting in improved productivity for computational science. In this project we will demonstrate novel techniques to facilitate efficient mapping of data objects, even partitioning individual variables, from the user space onto multiple storage tiers, and enable application-guided data reductions and transformations to address capacity and bandwidth bottlenecks. Our goal is to address the associated Input/ Output (I/O) and storage challenges in the context of current and emerging storage landscapes, and expedite insights into mission critical scientific processes.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Toward Transparent Data Management in Multi-layer Storage Hierarchy for HPC Systems

Upcoming exascale high performance computing (HPC) systems are expected to comprise multi-tier storage hierarchy, and thus will necessitate innovative storage and I/O mechanisms. Traditional disk and block-based interfaces and file systems face severe challenges in utilizing capabilities of storage hierarchies due to the lack of hierarchy support and semantic interfaces. Object-based and semantically-rich data abstractions for scientific data management on large scale systems offer a sustainable solution to these challenges. Such data abstractions can also simplify users involvement in data movement. Here, we take the first steps of realizing such an object abstraction and explore storage mechanisms for these objects to enhance I/O performance, especially for scientific applications. We explore how an object-based interface can facilitate next generation scalable computing systems by presenting the mapping of data I/O from two real world HPC scientific use cases: a plasma physics simulation code (VPIC) and a cosmology simulation code (HACC). Our storage model stores data objects in different physical organizations to support data movement across layers of memory/storage hierarchy. Our implementation sclaes well to 16K parallel processes, and compared to the state of the art, such as MPI-IO and HDF5, our object-based data abstractions and data placement strategy in multi-level storage hierarchy achieves up to 7 X I/O performance improvement for scientific data.

97 MATHEMATICS AND COMPUTING↗

Efficient Management of Complex Striped Files in Active Storage

Active Storage provides an opportunity for reducing the band- width requirements between the storage and compute elements of cur- rent supercomputing systems, and leveraging the processing power of the storage nodes used by some modern file systems. To achieve both objec- tives, Active Storage allows certain processing tasks to be performed directly on the storage nodes, near the data they manage. However, Active Storage must also support key requirements of scientific applications. In particular, Active Storage must be able to support striped files and files with complex formats (e.g., netCDF). In this paper, we describe how these important requirements can be addressed. The experimental results on a Lustre file system not only show that our proposal can re- duce the network traffic to near zero and scale the performance with the number of storage nodes, but also that it provides an efficient treatment of striped files and can manage files with complex data structures.

97 MATHEMATICS AND COMPUTING↗