Search NASA⌕ Search

SEARCH · Search NASA

Results for “data ingest”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

AI Curation Methods for NASA Scientific Data

The NASA Open Science Data Repository (OSDR) serves as a central hub for sharing and accessing NASA's vast collection of scientific data, supporting researchers across diverse fields. To enhance the efficiency, accuracy, and accessibility of this data, we are leveraging advanced artificial intelligence (AI) techniques as part of the AI for Curation project. By integrating large language models (LLMs) into our data curation workflow, we aim to streamline the entire process—from data submission to user interaction. This initiative focuses on improving key areas, including data ingestion, curation, and user engagement with curated datasets, impacting multiple domains and a wide user base. First, we are developing tools that can automatically parse data in various formats, using LLMs to convert unstructured data into structured, standardized formats. This reduces the manual effort required for curation, allowing curators to focus on more critical scientific analyses. Additionally, AI and machine learning (ML) models are being implemented to automate data validation and verification, ensuring the highest standards of data quality and reliability. Finally, we are creating a conversational AI agent to interact with the curated scientific studies in OSDR, helping users easily navigate the repository and access relevant data. By enhancing data discoverability and accessibility, these advancements will foster new research opportunities and promote the principles of open science.

Walter Alvarado↗

NASA GeneLab: Open Science for Life in Space

The NASA GeneLab project capitalizes on multi-omic technologies to maximize the return on spaceflight experiments. To do this, GeneLab maintains a publicly accessible database (GLDS) that houses spaceflight and spaceflight relevant multi-omics data and collaborates with NASA principal investigators and projects to generate additional omics data. GeneLab houses more than 350 transcriptomic, proteomic, metabolomic and epigenomic datasets from plant, animal and microbial experiments, with a growing number of these having been produced by the GeneLab Sequencing Lab. The GLDS contains rich metadata about each experiment and has integrated radiation dosimetry data from experiments flown on the Space Shuttle, International Space Station, and Free Flying spacecrafts. With the increasing amount and complexity of omics data being generated, GeneLab utilizes community-defined, common models for metadata and terminology so that omics data and results are discoverable and reliably reproducible. GeneLab uses the ISA-Tab specification and semantic model for organizing and representing omics metadata. In addition to metadata standards, data files must be open-source file or common exchange formats to ensure accessibility and usability by all users. To ease data ingestion and transfer, the web-based submission tool allows PIs a user-friendly user interface to curate, organize, and publish their space relevant omics data. In the more recent years, data curation and submission portal has incorporated the FAIR principles making data findable, accessible, interoperable, and reusable. To increase reusability of data, GeneLab has implemented an effort to present processed data in the GLDS in addition to the raw omics data. The processed data will enable interpretation of the data by a larger group of students, scientists and the general public. Standard pipelines for the transformation of raw data into visualizations were developed by four GeneLab Analysis Working Groups (animals, plants, microbes, multi-omics) comprised of over 200 scientists from NASA, industry, and academia. To explore the data, the GLDS provides users various tools for data analysis, collaborative workspace for file storage and sharing, and a visualization portal. The analysis platform built using the Galaxy toolshed provides access to a broad variety of users including those with limited bioinformatics experience and students to learn how to analyze spaceflight omics data. The visualization portal takes GeneLab one step closer to data democratization by removing all bioinformatics requisites to interpret transcriptomics data hosted in the repository. To train the next generation of scientists, NASA offers training programs such as GeneLab 4 High School (GL4HS) and GeneLab 4 Universities. NLM Curation at a Scale Workshop 2022 | NASA GeneLab (GL4U) to teach students bioinformatics and computational biology methods to analyze omics data. Discoveries made using GeneLab have begun and will continue to deepen our understanding of biology, advance the field of genomics, and help to discover cures for diseases, create better diagnostic tools, and ultimately allow astronauts to better withstand the rigors of long-duration spaceflight.

GeneLab↗

dCache - Keeping up With the Evolution of Science

The dCache project provides open-source software deployed internationally to satisfy ever more demanding storage requirements of various scientific communities. Its multifaceted approach provides an integrated way of supporting different use-cases with the same storage, from high throughput data ingest, through wide access and easy integration with existing systems, including event driven workflow management. With this presentation, we will show some of the recent developments that optimize data management and access to maximise the gain from stored data.

Mkrtchyan, Tigran↗

Beyond Fair: Engagement, Data Usability, and Open Community Productivity through the NASA Open Science Data Repository

The FAIR principle (findable, accessible, interoperable, and reusable) governs the storage and sharing of NASA space biology and health data[1]. These guiding principles maximize reuse of data and the reproducibility of scientific findings. The NASA Open Science Data Repository (OSDR; an expansion of NASA GeneLab) was built on the FAIR principles and houses over 500 studies and close to 1000 datasets from decades of space life sciences experiments. OSDR embodies the FAIR principles through data governance that includes mediated, embargoed, and fully open access data. The FAIR data governance principles were recently proposed to be expanded to encompass a FAIREST framework for assessing research data repositories (FAIR + Engagement, Social connections, and Trust)[2]. FAIREST emphasizes the importance of data repositories engaging with the scientific community and gaining the trust of researchers regarding data quality. Trust also refers to the TRUST principles developed for assessment of digital repositories: Transparency, Responsibility, User Focus, Sustainability, Technology[3]. We present the “Open Science for Life in Space” Analysis Working Groups (AWGs) as evidence regarding the power of engagement, social connections, and trust which has enhanced OSDR’s capabilities and productivity. AWG members engage in two main activities. One, members provide feedback on OSDR scientific standards for data ingestion, curation, and reuse (study, subject and assay metadata; processing pipelines; dataset formats and uniformed structures for machine-readability). Two, AWG members collaborate to mine-reuse OSDR data to conduct scientific analysis. With nearly 800 active members, the AWGs have resulted in 32 publications re-using OSDR data and contributed many papers in two major special issues in Cell (2020) and Nature (2024). AWGs also serve as networking groups, facilitate social connections between researchers at all levels of experience, and also have a social online ‘Forum’ used to keep members informed on projects and opportunities. This community-centric, productive, and trustworthy data culture has resulted in a broader effect with international space agencies, academics, and the commercial space sector wanting to submit their data to OSDR. Ten studies of Inspiration 4 data were recently publicly released by OSDR, as were some JAXA human data. Coming up soon in OSDR are data submissions from the European Space Agency, Virgin Galactic PIs, and SpaceX Polaris Dawn. A major benefit of OSDR is the array of standardized and uniformly formatted data (which was developed through AWG member consensus), from which visualization tools, analysis tools, and machine learning models can be built or trained. This talk will cover the Multi-Study Visualization Tool, the Environmental Data Application, RadLab, and a UCSF-NSF funded knowledge graph biomedical health discovery tool ‘SPOKE’ currently being integrated with OSDR. OSDR also provides training programs in bioinformatics and machine learning to improve the scientific community’s awareness of data availability and to boost their ability to perform data analysis. The increasing engagement of the scientific community and the public with technologies powered by artificial intelligence (AI) heightens the need for data analysis to be transparent. The AI for Life in Space initiative leverages the data products provided in OSDR to train AI models, with an emphasis on explainable and trustworthy AI, which would not be possible without FAIR data and metadata. Overall, here we will demonstrate the importance for NASA life sciences data repositories to adhere to the FAIREST framework, by providing examples and success stories from different aspects of OSDR.

data↗

Metadata Entry Optimization for NASA's Biological Institutional Scientific Collection (NBISC)

The NASA Biological Institutional Sample Collection (NBISC) at NASA’s Ames Research Center is a critical resource housing non-human samples collected from spaceflight missions and ground analog studies, primarily consisting of specimens from rats, mice, and select microbes. The primary objective of NBISC is to systematically receive, document, preserve, and facilitate access to these samples for the global scientific community. NBISC promotes international collaboration and maximizes the return on investment for precious tissues from spaceflight and analog experiments. Researchers can request physical samples through an online request form and subsequent written proposal review process. This study addresses two core research objectives: streamlining the NBISC sample lifecycle processes and strategizing for managing an influx of 50,000 tissue samples from a series of cosmic radiation analog experiments carried out at the NASA Space Radiation Laboratory (NSRL) by Drs. Eleanor Chang (Lawrence Berkeley Laboratory) and Polly Blakely (SRI). The Chang/Blakely studies investigated Harderian gland (HG) tumorigenesis in mice exposed to low dose and LET radiation comprising 8 different exposure protocols in over 4000 mice. NBISC sample metadata is stored in a Laboratory Information Management System (SLIMS). To streamline sample data entry, we customize python scripts using information extracted from the individual experimental protocols. The scripts automate entry into multiple SLIMS data fields including protocol name, unique sample barcode, tissue and sub-tissue information, freezer location, sample preservation method, etc. The semi-automated procedure significantly decreases the time spent on data entry by several orders of magnitude. Automation and data organization are essential, as they free up time for curation and promotion of the collection which, in turn, increase the accessibility of samples to the broader research community. NBISC benefits from streamlined data ingestion, and the methodologies developed here are applicable to other projects which use SLIMS including the NASA Biospecimen Sharing Program and GeneLab. As of Fall 2023, plans include transferring sample data from SLIMS to public facing repositories (OSDR and NLSP), expanding the reach of the Chang/Blakely sample collection. The Human Research Program Space Radiation Element plans to transfer non-human tissues from many more investigations to NBISC in the coming year.

Sample Repository↗

Metadata Entry Optimization For NASA's Biological Institutional Scientific Collection (NBISC)

The NASA Biological Institutional Sample Collection (NBISC) at NASA’s Ames Research Center is a critical resource housing non-human samples collected from spaceflight missions and ground analog studies, primarily consisting of specimens from rats, mice, and select microbes. The primary objective of NBISC is to systematically receive, document, preserve, and facilitate access to these samples for the global scientific community. NBISC promotes international collaboration and maximizes the return on investment for precious tissues from spaceflight and analog experiments. Researchers can request physical samples through an online request form and subsequent written proposal review process. This study addresses two core research objectives: streamlining the NBISC sample lifecycle processes and strategizing for managing an influx of 50,000 tissue samples from a series of cosmic radiation analog experiments carried out at the NASA Space Radiation Laboratory (NSRL) by Drs. Eleanor Chang (Lawrence Berkeley Laboratory) and Polly Blakely (SRI). The Chang/Blakely studies investigated Harderian gland (HG) tumorigenesis in mice exposed to low dose and LET radiation comprising 8 different exposure protocols in over 4000 mice. NBISC sample metadata is stored in a Laboratory Information Management System (SLIMS). To streamline sample data entry, we customize python scripts using information extracted from the individual experimental protocols. The scripts automate entry into multiple SLIMS data fields including protocol name, unique sample barcode, tissue and sub-tissue information, freezer location, sample preservation method, etc. The semi-automated procedure significantly decreases the time spent on data entry by several orders of magnitude. Automation and data organization are essential, as they free up time for curation and promotion of the collection which, in turn, increase the accessibility of samples to the broader research community. NBISC benefits from streamlined data ingestion, and the methodologies developed here are applicable to other projects which use SLIMS including the NASA Biospecimen Sharing Program and GeneLab. As of Fall 2023, plans include transferring sample data from SLIMS to public facing repositories (OSDR and NLSP), expanding the reach of the Chang/Blakely sample collection. The Human Research Program Space Radiation Element plans to transfer non-human tissues from many more investigations to NBISC in the coming year.

Biospecimen↗

Metall: A persistent memory allocator for data-centric analytics

Data analytics applications transform raw input data into analytics-specific data structures before performing analytics. Unfortunately, such data ingestion steps are often more expensive than analytics. In addition, various types of NVRAM devices are already used in many HPC systems today. Such devices will be useful for storing and reusing data structures beyond a single process life cycle. We developed Metall, a persistent memory allocator built on top of the memory-mapped file mechanism. Metall enables applications to transparently allocate custom C++ data structures into various types of persistent memories. Metall incorporates a concise and high-performance memory management algorithm inspired by Supermalloc and the rich C++ interface developed by Boost.Interprocess library. On a dynamic graph construction workload, Metall achieved up to 11.7x and 48.3x performance improvements over Boost.Interprocess and memkind (PMEM kind), respectively. We also demonstrate Metall’s high adaptability by integrating Metall into a graph processing framework, GraphBLAS Template Library. Here this study’s outcomes indicate that Metall will be a strong tool for accelerating future large-scale data analytics by allowing applications to leverage persistent memory efficiently.

97 MATHEMATICS AND COMPUTING↗

Storage for Advanced Scientific Use-Cases and Beyond

The dCache project provides open-source storage software deployed internationally to satisfy ever more demanding scientific storage requirements. Its multifaceted approach provides an integrated way of supporting different use-cases with the same storage, from high throughput data ingest, through wide access and easy integration with existing systems. In this paper, we describe some of the recent features that facilitate the use of storage to maximise the gain from stored data, including quality-of-service management, heterogeneous systems - both through integrated tertiary storage support and geographical locality - the parallel NFS (pNFS) extension, and innovative delegated authorisation schemes.

43 PARTICLE ACCELERATORS↗

DER Cybersecurity Detection and Response Suite

SAND2024-08475O The Distributed Energy Resource (DER) Cybersecurity Detection and Response Suite is a solution for distributed energy resource (DER) systems. The DER Security Orchestration, Automation, and Response (SOAR) solution that uses alerts from signature- and behavior-based Intrusion Detection Systems are intended to be deployed as bump-in-the-wire (BITW) devices in front of DER equipment. The fielded application would use multiple intrusion detection systems that report data to SOAR to respond to cyberattacks. The suite consists of two software components: • The proactive intrusion detection and mitigation system (PIDMS) secures grid-edge photovoltaic smart inverters and other equipment in distributed energy resource systems. It is a distributed BITW solution; cyber and physical data are automatically processed using network inspection tools and custom machine learning algorithms to detect abnormal events and correlate cyber-physical events. • The Security Orchestration, Automation, and Response for Distributed Energy Resources (SOAR4DER) application ingests data from several intrusion detection systems to quickly block attacks and revert DER systems to good states. Using a collection of intrusion detection system technologies on a BITW device, it incorporates physical and cyber data to detect abnormal and potential malicious behaviors. Multiple SOAR playbooks then use the intrusion detection system data streams to automatically defend the system. SOAR4DER system testing showed detection and response times under 30 seconds for all adversary reconnaissance, denial-of-service attacks, malicious Modbus commands, brute-force logins, and machine-in-the-middle attacks. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Johnson, Jay↗

Computational Reproducibility of Scientific Workflows at Extreme Scales

We propose an approach for improved reproducibility that includes capturing and relating provenance characteristics and performance metrics, in a hybrid queriable system, the ProvEn server. The system capabilities are illustrated on two use cases: scientific reproducibility of results in the ACME climate simulations and performance reproducibility in molecular dynamics workflows on HPC computing platforms.

provenance, Reproducibility, Climate & Earth Syste↗

SpaceCube 2.0: An Advanced Hybrid Onboard Data Processor

The SpaceCube 2.0 is a compact, high performance, low-power onboard processing system that takes advantage of cutting-edge hybrid (CPU/FPGA/DSP) processing elements. The SpaceCube 2.0 design concept includes two commercial Virtex-5 field-programmable gate array (FPGA) parts protected by gradiation hardened by software" technology, and possesses exceptional size, weight, and power characteristics [5x5x7 in., 3.5 lb (approximately equal to 12.7 x 12.7 x 17.8 cm, 1.6 kg) 5-25 W, depending on the application fs required clock rate]. The two Virtex-5 FPGA parts are implemented in a unique back-toback configuration to maximize data transfer and computing performance. Draft computing power specifications for the SpaceCube 2.0 unit include four PowerPC 440s (1100 DMIPS each), 500+ DSP48Es (2x580 GMACS), 100+ LVDS high-speed serial I/Os (1.25 Gbps each), and 2x190 GFLOPS single-precision (65 GFLOPS double-precision) floating point performance. The SpaceCube 2.0 includes PROM memory for CPU boot, health and safety, and basic command and telemetry functionality; RAM memory for program execution; and FLASH/EEPROM memory to store algorithms and application code for the CPU, FPGA, and DSP processing elements. Program execution can be reconfigured in real time and algorithms can be updated, modified, and/or replaced at any point during the mission. Gigabit Ethernet, Spacewire, SATA and highspeed LVDS serial/parallel I/O channels are available for instrument/sensor data ingest, and mission-unique instrument interfaces can be accommodated using a compact PCI (cPCI) expansion card interface. The SpaceCube 2.0 can be utilized in NASA Earth Science, Helio/Astrophysics and Exploration missions, and Department of Defense satellites for onboard data processing. It can also be used in commercial communication and mapping satellites.

Lin, Michael↗

Surface Biology & Geology Pathfinder Data Analysis Pipeline

NASA's future global orbital mission, currently in development as the Surface Biology and Geology (SBG) Designated Observable study, will acquire relatively high resolution solar-reflected spectroscopy and thermal infrared observations. Innovative processes must be utilized for handling the high volume of data anticipated to be collected, which is anticipated to exceed 100 terabytes/day, greater than NASA's total extant airborne hyperspectral data collection. Collecting, processing/re-processing, disseminating, and exploiting this volume of data presents new challenges. To begin addressing them, NASA is drawing upon the expertise developed from its astrophysics programs to address Earth science and applications. Specifically, NASA is adapting the science processing operations technology developed for the Kepler and TESS planet-hunting missions for imaging spectroscopy data processing. This technology development has been the foundation for the remarkable scientific successes of Kepler and TESS. The Kepler/TESS data processing technology provides a scalable architecture for robust, repeatable, and replicable science and application products while enabling the Earth science community to develop, test, and implement new algorithms. Our effort to leverage this existing capability has begun by ingesting data and applying workflows from the EO-1/Hyperion 17-year mission archive that provides globally sampled visible through shortwave infrared spectra that are representative of SBG data types and volumes. This pathfinding data processing system will help define the solutions to processing SBG data volumes and will enable the scientific community to interact with the data and processing pipeline to create new science products.

Jenkins, Jon↗

The Winter 2010 and 2011 FRONT/NIRSS In-Flight Icing Hazard Detection Project

The NASA Icing Remote Sensing System (NIRSS) deploys a vertically-pointing K-band radar, a lidar ceiliometer, and a profiling microwave radiometer to obtain measurements for diagnosing local inflight icing conditions. RAL is working with NASA GRC to develop algorithms and data ingest and display software for the system. NASA has an ongoing activity to develop remote sensing technologies for the detection and measurement of icing conditions aloft. As part of that effort NASA teamed with NCAR to develop software that fuses data from multiple instruments into a single detected icing condition product. The multiple instrument approach, which is the current emphasis of this activity, utilizes a K-band vertical staring radar, a microwave radiometer that detects twelve frequencies between 22 and 59 GHz, and a lidar ceilometer. The radar data determine cloud boundaries, the radiometer determines the sub-freezing temperature heights and total liquid water content, and the ceilometer refines the lower cloud boundary. Data is post-processed in C++ program with a Java-based web display of resultant supercooled LWC profile and aircraft hazard identification. In 2010, a multi-channel scanning radiometer, designed and built by Radiometrics, Inc. under a SBIR grant,,was added to the system to assess its utility in improving icing diagnoses.

Serke, David↗

Optical mass memory system (AMM-13). AMM-13 system segment specification

The performance, design, development, and test requirements for an optical mass data storage and retrieval system prototype (AMM-13) are established. This system interfaces to other system segments of the NASA End-to-End Data System via the Data Base Management System segment and is designed to have a storage capacity of 10 to the 13th power bits (10 to the 12th power bits on line). The major functions of the system include control, input and output, recording of ingested data, fiche processing/replication and storage and retrieval.

Bailey, G. A.↗

Meteorological Observations Available for the State of Utah

The National Weather Service’s Meteorological Assimilation Data Ingest System (MADIS) contains a large number of station networks of surface and upper air meteorological observations for the state of Utah. In addition to MADIS, observations from individual station networks may also be available. It has been confirmed that LLNL has access to the data sources listed below.

54 ENVIRONMENTAL SCIENCES↗

Technical Note: NASAaccess – A Tool for Access, Reformatting, and Visualization of Remotely Sensed Earth Observation and Climate Data

The National Aeronautics and Space Administration (NASA) has launched a new initiative, the Open-Source Science Initiative (OSSI), to enable and support science towards openness. The OSSI supports open-source software development and dissemination. In this work, we present NASAaccess, which is an open-source software package and web-based environmental modeling application for earth observation data accessing, reformatting, and presenting quantitative data products. The main objective of developing the NASAaccess platform is to facilitate exploration, modeling, and understanding of earth data for scientists, stakeholders, and concerned citizens whose objectives align with the new OSSI goals. The NASAaccess platform is available as software packages (i.e., the R and conda packages) as well as an interactive-format web-based environmental modeling application for earth observation data developed with Tethys Platform. NASAaccess has been envisioned as lowering the technical barriers and simplifying the process of accessing scalable distributed computing resources and leveraging additional software for data and computationally intensive modeling frameworks. Specifically, NASAaccess has been developed to meet the need for seamless earth observation remote-sensing and climate data ingestion into various hydrological modeling frameworks. Moreover, NASAaccess is also contributing to keeping interested parties and stakeholders engaged with environmental modeling, accessing the information available in various remote-sensing products. NASAaccess' current capabilities cover various NASA datasets and products that include the Global Precipitation Measurement (GPM) data products, the Global Land Data Assimilation System (GLDAS) land surface states and fluxes, and the NASA Earth Exchange Global Daily Downscaled Projections (NEX-GDDP) Coupled Model Intercomparison Project Phase 5 (CMIP5) and Coupled Model Intercomparison Project Phase 6 (CMIP6) climate change dataset products.

Ibrahim Nourein Mohammed↗

Crew Health and Performance Integrated Data Architecture (CHP-IDA) Project

BACKGROUND: Future Human Exploration missions introduce a new paradigm as crews move further from the resupply and near real-time ground support typical of Low Earth Orbit missions today. Without immediate support from ground-based personnel, exploration crews will be more reliant on inflight data and technology to respond to emergencies and anomalies. A data architecture to support a new generation of technologies, employing advanced analytical and predictive modeling techniques, is needed to enable crew autonomy. OVERVIEW: The Crew Health and Performance Integrated Data Architecture (CHP-IDA) project funded by NASA’s Exploration Medical Integrated Product Team (XMIPT) is laying a foundation for future in-flight informatics by providing a back-end architecture for collecting, storing, and integrating multiple sources of data generated by and around the crew. CHP-IDA provides a platform for common data models and Application Programming Interfaces to access, integrate, process, and display CHP data (e.g., environmental, exercise, medical, sleep, performance, etc.). This will facilitate the increased situation awareness and decision support required by the crew and remote support of exploration missions. This presentation will describe the currently ongoing effort to develop and evaluate a path-to-flight concept of the CHP-IDA software and its core capabilities. Current integrations will be discussed, including analytics for Extravehicular Activity metabolic rate and data ingestion from a multi-functional integrated medical device. The presentation will also provide examples of scenarios used to demonstrate the CHP-IDA through human-in-the-loop test bed activities as well as examples of appropriate system performance metrics. DISCUSSION: Today, in-flight data is often siloed, unsynchronized, and largely inaccessible in real time. Many data sets require manual entry and/or data transfer between vehicles and the ground. These issues contribute to risks in supporting exploration medical capabilities. The CHP-IDA is a back-end data system providing core capabilities needed for timely and meaningful data insights across CHP domains to crew and remote personnel to enable increased crew autonomy. Future work includes collaboration with additional CHP domains, new technology integrations, and further demonstrations of the IDA within different vehicle and communication latency contexts. LEARNING OBJECTIVES 1. The audience will understand that the CHP-IDA is a back-end system, providing a platform to facilitate access, promote decision tools, and provide meaningful insights to crew and to remote stakeholders during exploration missions. 2. The audience will gain insight into human-centered research and activities used to discover CHP domain data needs and pain points and how this information is used to guide development of the IDA.

Exploration↗

MyEMSL

MyEMSL provides the following features: user-assisted upload of data from scientific instruments to an archive, tracking data though the upload process, providing metadata for recovery of uploaded data, providing users with access to the data. These features are supplied by a mixture of open source software and internally developed software. MyESML also provides a notification system that allows the system to communicate with users and other parts of the MyEMSL system. This allows the system to support automated workflows and user interaction. Reprocessing of already ingested data is also supported.

Glass, Kevin↗