Search NASA⌕ Search

SEARCH · Search NASA

Results for “Scientific Data Management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

SIRIUS: Science-Driven Data Management for Multi-Tiered Storage

The data sets being generated by large applications on very large-scale systems are increasing in both size and complexity. At the same time, there are new ways available to store and access these data sets. The goal in this project is to develop software that applications can use to make use of new and existing storage technologies in more sophisticated ways. One challenge in scientific data management is handling ‘hot’ vs ‘cold’ data. Data that is hot is data that is needed (or will be needed soon) in order for the program to continue progressing, while cold data is either output (and so will not be need further during the life of the program) or will not be needed until significantly later in the program’s run. Hot data should be stored in a way that allows fast access. On most systems, economic factors lead to an inverse relationship between storage performance and storage capacity and so fast access storage is limited. This makes it important to correctly place hot and cold data and avoid cold data unnecessarily consuming precious resources. In this reporting period, we addressed this challenge in various ways and at various levels. Data management frameworks offer only limited control to applications in how data is stored. We have added software capabilities for seamlessly moving data between layers of the storage technology using promote and demote functions to existing software frameworks. This gives direct control to applications in deciding what priority data receives. Additionally, we integrated different storage layer management frameworks in order to allow data to be exchanged and moved between storage layers in a consistent way across the application. Further, applications are not always able to directly decide what storage level makes sense for a given piece of data without an understanding of the underlying storage technologies. Data storage frameworks are often positioned to make these sorts of decisions in service of the application. We have added machine-learning based capabilities to data staging frameworks in order to make intelligent decisions about where data should be stored given learning about patterns in previous usage of similar data.

97 MATHEMATICS AND COMPUTING↗

A comprehensive and fair comparison of two neural operators (with practical extensions) based on $\mathrm{FAIR}$ data

Neural operators can learn nonlinear mappings between function spaces and offer a new simulation paradigm for real-time prediction of complex dynamics for realistic diverse applications as well as for system identification in science and engineering. Herein, we investigate the performance of two neural operators, which have shown promising results so far, and we develop new practical extensions that will make them more accurate and robust and importantly more suitable for industrial-complexity applications. The first neural operator, DeepONet, was published in 2019 (Lu et al., 2019), and its original architecture was based on the universal approximation theorem of Chen & Chen (1995). The second one, named Fourier Neural Operator or FNO, was published in 2020, and it is based on parameterizing the integral kernel in the Fourier space. DeepONet is represented by a summation of products of neural networks (NNs), corresponding to the branch NN for the input function and the trunk NN for the output function; both NNs are general architectures, e.g., the branch NN can be replaced with a CNN or a ResNet. According to Kovachki et al. (2021), FNO in its continuous form can be viewed conceptually as a DeepONet with a specific architecture of the branch NN and a trunk NN represented by a trigonometric basis. In order to compare FNO with DeepONet computationally for realistic setups, we develop several extensions of FNO that can deal with complex geometric domains as well as mappings where the input and output function spaces are of different dimensions. We also develop an extended DeepONet with special features that provide inductive bias and accelerate training, and we present a faster implementation of DeepONet with cost comparable to the computational cost of FNO, which is based on the Fast Fourier Transform. Here we consider 16 different benchmarks to demonstrate the relative performance of the two neural operators, including instability wave analysis in hypersonic boundary layers, prediction of the vorticity field of a flapping airfoil, porous media simulations in complex-geometry domains, etc. We follow the guiding principles of FAIR (Findability, Accessibility, Interoperability, and Reusability) for scientific data management and stewardship. The performance of DeepONet and FNO is comparable for relatively simple settings, but for complex geometries the performance of FNO deteriorates greatly. We also compare theoretically the two neural operators and obtain similar error estimates for DeepONet and FNO under the same regularity assumptions.

42 ENGINEERING↗

Dynamo: Handling Scientific Data Across Sites and Storage Media

Abstract Dynamo is a full-stack software solution for scientific data management. Dynamo’s architecture is modular, extensible, and customizable, making the software suitable for managing data in a wide range of installation scales, from a few terabytes stored at a single location to hundreds of petabytes distributed across a worldwide computing grid. This article documents the core system design of Dynamo and describes the applications that implement various data management tasks. A brief report is also given on the operational experiences of the system at the CMS experiment at the CERN Large Hadron Collider and at a small-scale analysis facility.

Iiyama, Yutaro (ORCID:0000000282975930)↗

Foundations of machine learning for low-temperature plasmas: methods and case studies

Abstract Machine learning (ML) and artificial intelligence have proven to be an invaluable tool in tackling a vast array of scientific, engineering, and societal problems. The main drivers behind the recent proliferation of ML in practically all aspects of science and technology can be attributed to: (a) improved data acquisition and inexpensive data storage; (b) exponential growth in computing power; and (c) availability of open-source software and resources that have made the use of state-of-the-art ML algorithms widely accessible. The impact of ML on the field of low-temperature plasmas (LTPs) could be particularly significant in the emerging applications that involve plasma treatment of complex interfaces in areas ranging from the manufacture of microelectronics and processing of quantum materials, to the LTP-driven electrification of the chemical industry, and to medicine and biotechnology. This is primarily due to the complex and poorly-understood nature of the plasma-surface interactions in these applications that pose unique challenges to the modeling, diagnostics, and predictive control of LTPs. As the use of ML is becoming more prevalent, it is increasingly paramount for the LTP community to be able to critically analyze and assess the concepts and techniques behind data-driven approaches. To this end, the goal of this paper is to provide a tutorial overview of some of the widely-used ML methods that can be useful, amongst others, for discovering and correlating patterns in the data that may be otherwise impractical to decipher by human intuition alone, for learning multivariable nonlinear data-driven prediction models that are capable of describing the complex behavior of plasma interacting with interfaces, and for guiding the design of experiments to explore the parameter space of plasma-assisted processes in a systematic and resource-efficient manner. We illustrate the utility of various supervised, unsupervised and active learning methods using LTP datasets consisting of commonly-available, information-rich measurements (e.g. optical emission spectra, current–voltage characteristics, scanning electron microscope images, infrared surface temperature measurements, Fourier transform infrared spectra). All the ML demonstrations presented in this paper are carried out using open-source software; the datasets and codes are made publicly available. The FAIR guiding principles for scientific data management and stewardship can accelerate the adoption and development of ML in the LTP community.

Physics↗

Distributed and cloud-based multi-model analytics experiments on large volumes of climate change data in the earth system grid federation eco-system

A case study on climate models intercomparison data analysis addressing several classes of multi-model experiments is being implemented in the context of the EU H2020 INDIGO-DataCloud project. Such experiments require the availability of large amount of data (multi-terabyte order) related to the output of several climate models simulations as well as the exploitation of scientific data management tools for large-scale data analytics. More specifically, the paper discusses in detail a use case on precipitation trend analysis in terms of requirements, architectural design solution, and infrastructural implementation. The experiment has been tested and validated on CMIP5 datasets, in the context of a large scale distributed testbed across EU and US involving three ESGF sites (LLNL, ORNL, and CMCC) and one central orchestrator site (PSNC).

Fiore, S.↗

Extending Skel to Support the Development and Optimization of Next Generation I/O Systems

As the memory and storage hierarchy get deeper and more complex, it is important to have new benchmarks and evaluation tools that allow us to explore the emerging middleware solutions to use this hierarchy. Skel is a tool aimed at automating and refining this process of studying HPC I/O performance. It works by generating application I/O kernel/benchmarks as determined by a domain-specific model. This paper provides some techniques for extending Skel to address new situations and to answer new research questions. For example, we document use cases as diverse as using Skel to troubleshoot I/O performance issues for remote users, refining an I/O system model, and facilitating the development and testing of a mechanism for runtime monitoring and performance analytics. We also discuss data oriented extensions to Skel to support the study of compression techniques for Exascale scientific data management.

97 MATHEMATICS AND COMPUTING↗

High performance multivariate visual data exploration for extremely large data

One of the central challenges in modern science is the need to quickly derive knowledge and understanding from large, complex collections of data. We present a new approach that deals with this challenge by combining and extending techniques from high performance visual data analysis and scientific data management. This approach is demonstrated within the context of gaining insight from complex, time-varying datasets produced by a laser wakefield accelerator simulation. Our approach leverages histogram-based parallel coordinates for both visual information display as well as a vehicle for guiding a data mining operation. Data extraction and subsetting are implemented with state-of-the-art index/query technology. This approach, while applied here to accelerator science, is generally applicable to a broad set of science applications, and is implemented in a production-quality visual data analysis infrastructure. We conduct a detailed performance analysis and demonstrate good scalability on a distributed memory Cray XT4 system.

Rübel, O↗

A Study of the Trade-off Between Reducing Precision and Reducing Resolution for Data Analysis and Visualization

There presently exist two dominant strategies to reduce data sizes in analysis and visualization: reducing the precision of the data, e.g., through quantization, or reducing its resolution, e.g., by subsampling. Both have advantages and disadvantages and both face fundamental limits at which the reduced information ceases to be useful. The paper explores the additional gains that could be achieved by combining both strategies. In particular, we present a common framework that allows us to study the trade-off in reducing precision and/or resolution in a principled manner. We represent data reduction schemes as progressive streams of bits and study how various bit orderings such as by resolution, by precision, etc., impact the resulting approximation error across a variety of data sets as well as analysis tasks. Additionally, we compute streams that are optimized for different tasks to serve as lower bounds on the achievable error. Scientific data management systems can use the results proposed in this paper as guidance on how to store and stream data to make efficient use of the limited storage and bandwidth in practice.

97 MATHEMATICS AND COMPUTING↗

Simulation INsight and Analysis

Sina is a tool set for modern scientific data management that provides flexible, light-weight support of non-bulk data capture for retention in and queries against SQL and noSQL data stores. HPC simulations traditionally maintain their data in files. Extracting data of interest for subsequent analysis then requires the time-consuming process of traversing directories and scraping data from files in a variety of formats. Sina facilitates capturing relevant data during execution or post-processing of simulation runs for retention in and queries from a modern data store. The tools are sufficiently general to allow for the inclusion of new fields as scientists learn more about their data. Libraries, currently in C++ and Python, and a command line interface (CLI) are provided. Sina's flexibility starts with a general schema, in JSON, for the collection of non-bulk simulation data. JSON provides a flexible, human-readable representation of the data that of interest. Sina currently has a C++ library for simulations to write data to and read from a schema-compliant file for subsequent ingestion into one of the supported data stores. However, applications are free to write their data directly into a schema-compliant file. Python packages provide data ingestion, management, query, and export capabilities. A command line interface (CLI) provides simplified access to these features. A common application programming interface (API) is used to maintain and query data in any of the supported data stores, which are currently limited to SQL and Apache Cassandra (a column store). Tutorials, demonstrations, and examples illustrate aspects of the process using scripts and Jupyter notebooks.

Pauli, EstebanT↗

ornladios/ADIOS2

ADIOS 2: The Adaptable Input Output (I/O) System version 2 is an open-source framework that addresses scientific data management challenges, e.g. scalable parallel I/O, as we approach the exascale era in high-performance computing (HPC). ADIOS 2 bindings are available in C++, C, Fortran, Python and can be used on supercomputers, personal computers, and cloud systems running on Linux, macOS and Windows. ADIOS 2 has out-of-the-box support for MPI and serial environments.

ECP↗

Open Source Scalable Data Services and Data Fusion for Biological and Environmental Sciences (SBIR Phase I Final Scientific/ Technical Report)

The overarching goal of the project is to develop an integrated open-source scientific data management system (Apache V2 license), ResonantEco, that meets the need of biological and environmental researchers and developers for data management, curation, and data processing for analyses with a wide range of scale and complexity. ResonantEco will provide web enabled data services with features such as unified data interfaces and federated views of data and metadata for heterogeneous data sources with an interactive web client for data exploration. Our use of the term fusion is taken from geospatial (GIS) domain where data fusion is often synonymous with data integration. In particular, data integration in ResonantEco involves combining data residing in different sources and providing users with a unified view of them.

99 GENERAL AND MISCELLANEOUS↗

From Raw to Curated Data: A Lakehouse Approach for Scientific Workflows

This report provides a technical overview of how to go from raw to curated data in three stages using a lakehouse approach. We focus on the application of open source tools in scientific use cases (while noting parallels to enterprise and commercial alternatives). Our goal is to provide scientific data managers and infrastructure providers with a common frame of reference for understanding and applying modern lakehouse technologies and approaches.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

A KBase Case Study on Genome-wide Transcriptomics and Plant Primary Metabolism in Sorghum

A better understanding of the genetic and metabolic mechanisms that confer stress resistance and tolerance in plants is key to engineering new crops through advanced breeding technologies. This requires a systems biology approach that builds on a genome-wide understanding of the regulation of gene expression, plant metabolism, physiology and growth. In this study, we examine the response to drought stress in Sorghum, as we leverage the tools for transcriptomics and plant metabolic modeling we have implemented at the U.S. Department of Energy Systems Biology Knowledgebase (KBase). KBase enables researchers worldwide to collaborate and advance research by allowing them to upload private or public data into the KBase Narrative Interface, empowering them to analyze this data using a rich, extensible array of computational and data-analytics tools, and allows them to securely share scientific workflows and conclusions. We demonstrate how to use the current RNA-seq tools in KBase, applicable to both plants and microbes, to assemble and quantify long transcripts and identify differentially expressed genes effectively. More specifically, we demonstrate the utility of the platform by identifying key genes that are differentially expressed during drought-stress in Sorghum bicolor, which is an important sustainable production crop plant. We then show how to use KBase tools to predict the membership of genes in metabolic pathways and examine expression data in the context of metabolic subsystems. We demonstrate the power of the platform by making the data analysis and interpretation available to the biologists in the reproducible, re-usable, point-and-click format of a KBase Narrative thus promoting FAIR (Findable, Accessible, Interoperable and Reusable) guiding principles for scientific data management and stewardship.

59 BASIC BIOLOGICAL SCIENCES↗

The Solar Optical Telescope /SOT/

The Solar Optical Telescope (SOT) is a 1-m class, high resolution solar telescope which NASA plans to operate on the Shuttle Spacelab during the mid and late 1980's. SOT will provide resolution down to 0.1 arcsec, and can therefore be used to solve many problems of solar atmosphere structure and dynamics. In particular, SOT should provide definite data on (1) the source of the sun's magnetic field, (2) the amount of power in waves generated in the photosphere, (3) the heating of the chromosphere, and (4) the local and global mass and energy balance in the solar transition region. The SOT Scientific Working Group will help plan, develop, and update the overall scientific plan for the SOT, as well as make recommendations with regard to the SOT project, instrumentation optimization, and scientific data management.

Jordan, S. D.↗

Pilot climate data system user's guide

Instructions for using the Pilot Climate Data System (PCDS), an interactive, scientific data management system for locating, obtaining, manipulating, and displaying climate-research data are presented. The PCDS currently provides this supoort for approximately twenty data sets. Figures that illustrate the terminal displays which a user sees when he/she runs the PCDS and some examples of the output from this system are included. The capabilities which are described in detail allow a user to perform the following: (1) obtain comprehensive descriptions of a number of climate parameter data sets and the associated sensor measurements from which they were derived; (2) obtain detailed information about the temporal coverage and data volume of data sets which are readily accessible via the PCDS; (3) extract portions of a data set using criteria such as time range and geographic location, and output the data to tape, user terminal, system printer, or online disk files in a special data-set-independent format; (4) access and manipulate the data in these data-set-independent files, performing such functions as combining the data, subsetting the data, and averaging the data; and (5) create various graphical representations of the data stored in the data-set-independent files.

Reph, M. G.↗

NASA Tech Briefs, April 2013

Topics covered include: Fully Integrated, Miniature, High-Frequency Flow Probe Utilizing MEMS Leadless SOI Technology; Nanoscale Surface Plasmonics Sensor With Nanofluidic Control; Advanced Dispersed Fringe Sensing Algorithm for Coarse Phasing Segmented Mirror Telescopes; Neural Network Back-Propagation Algorithm for Sensing Hypergols; Bulk Moisture and Salinity Sensor; Change-Based Satellite Monitoring Using Broad Coverage and Targetable Sensing; Circularly Polarized Microwave Antenna Element with Very Low Off-Axis Cross-Polarization; Ultra-Low Heat-Leak, High-Temperature Superconducting Current Leads for Space Applications; Flash Cracking Reactor for Waste Plastic Processing; An Automated Safe-to-Mate (ASTM) Tester; Wireless Chalcogenide Nanoionic-Based Radio-Frequency Switch; Compute Element and Interface Box for the Hazard Detection System; DOT Transmit Module; Composite Aerogel Multifoil Protective Shielding; Li-Ion Electrolytes with Improved Safety and Tolerance to High-Voltage Systems; Polymer-Reinforced, Non-Brittle, Lightweight Cryogenic Insulation; Controlled, Site-Specific Functionalization of Carbon Nanotubes with Diazonium Salts; Regenerable Sorbent for CO2 Removal; Sprayable Aerogel Bead Compositions With High Shear Flow Resistance and High Thermal Insulation Value; Lexan Linear Shaped Charge Holder with Magnets and Backing Plate; Robotic Ankle for Omnidirectional Rock Anchors; Wind, Wave, and Tidal Energy Without Power Conditioning; An Active Heater Control Concept to Meet IXO Type Mirror Module Thermal-Structural Distortion Requirement; Waterless Clothes-Cleaning Machine; Integrated Electrical Wire Insulation Repair System; LVGEMS Time-of-Flight Mass Spectrometry on Satellites; Surface Inspection Tool for Optical Detection of Surface Defects; Per-Pixel, Dual-Counter Scheme for Optical Communications; Certification-Based Process Analysis; Surface Navigation Using Optimized Waypoints and Particle Swarm Optimization; Smart-Divert Powered Descent Guidance to Avoid the Backshell Landing Dispersion Ellipse; Estimating Foreign-Object-Debris Density from Photogrammetry Data; Adaptive Sampling of Spatiotemporal Phenomena with Optimization Criteria; Building a 2.5D Digital Elevation Model From 2D Imagery; Eyes on the Earth 3D; Target Trailing With Safe Navigation for Maritime Autonomous Surface Vehicles; Adams-Based Rover Terramechanics and Mobility Simulator - ARTEMIS; ISTP CDF Skeleton Editor; Uplink Summary Generator (ULSGEN) Version 1.0; Robotics On-Board Trainer (ROBoT); Software Engineering Tools for Scientific Models; Automatic Data Filter Customization Using a Genetic Algorithm; Tracker Toolkit; Towards Efficient Scientific Data Management Using Cloud Storage; On a Formal Tool for Reasoning About Flight Software Cost Analysis; A Nanostructured Composites Thermal Switch Controls Internal and External Short Circuit in Lithium Ion Batteries; Spacecraft Crew Cabin Condensation Control; and Functional Near-Infrared Spectroscopy Signals Measure Neuronal Activity in the Cortex.

Source record↗

INCREASING THE TRANSPARENCY AND REPRODUCIBILITY OF SPACE RADIATION SCIENCE: THE RADIATION BIOLOGY ONTOLOGY

Among the primary objectives of the Open/Open-Source Science paradigm are making scientific investigation data transparent and results reproducible [1], objectives shared by the FAIR principles [2]. To accomplish this, the conceptual framework that includes all the investigation objects needs to be accurately captured and communicated to all data consumers. A large part of this requires using metadata standards to annotate data collected. These standards should be readily accessible, informed by scientific community consensus and sufficiently specific to encompass all of the important aspects of the investigation. Starting in 2020 we have been co-leading an open consortium to develop a new metadata standard, the Radiation Biology Ontology (RBO), through the Open Biological and Biomedical Ontologies (OBO) Foundry [3]. We began by transforming many of the terms from the National Council on Radiation Protection and Measurement into concepts that can be formally related to existing OBO Foundry classes or attributes. We then identified and imported into the RBO existing OBO Foundry classes that have obvious relevance for radiation biomedicine (for example, concepts from the Environment Ontology that describe radiative processes, and concepts from the Gene Ontology dealing with molecular and cellular responses to radiation). Finally, we scrutinized datasets from investigations of radiation effects held in NASA GeneLab and LSDA repositories and added additional classes, instances, and attributes into the RBO that should be used to annotate these data. We developed the RBO using the open-source tools of GitHub and publish the RBO periodically through the NIH/NCBI BioPortal website, so systems worldwide can leverage the knowledge it contains [4]. This initial phase of concept modeling has yielded an RBO that at present has more than 300 declared concepts, with more than 3500 additional concepts imported from other OBO Foundry ontologies. While this first phase has focused on concepts for annotating samples, environments, exposures, and measurements, the next phase will center on supporting annotation of results and findings, such as concept models of molecular, cellular and tissue effects. The value of the RBO will be determined in part by our ability to engage the community in its development, and we have established a Radiobiology Informatics Consortium with unrestricted membership as the owner of the RBO in order to encourage investigators, system owners and other to join in this effort. Anyone can report issues or request new concept modeling or other features directly on GitHub. By using the BioPortal application programming interface, systems can pose dynamic queries to the latest version of the RBO for information on individual classes or entire hierarchies; this design eliminates the need for systems to be updated in order to use newer versions of the RBO. We hope to contribute to the advancement of open radiobiological science through the continued, open development of the RBO, that will provide more precise, machine-interpretable descriptions of investigations, as well as support data meta-analysis through machine learning or other artificial intelligence methods. REFERENCES [1] Open science in space. Nature Medicine, 2021. 27(9): p. 1485-1485. [2] Wilkinson, M.D., et al., The FAIR Guiding Principles for scientific data management and stewardship. Sci Data, 2016. 3: p. 160018. [3] Smith, B., et al., The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration. Nat Biotechnol, 2007. 25(11): p. 1251-5. [4] Whetzel, P.L., et al., BioPortal: enhanced functionality via new Web services from the National Center for Biomedical Ontology to access and use ontologies in software applications. Nucleic Acids Res, 2011. 39(Web Server issue): p. W541-5.

informatics↗