Search NASA⌕ Search

SEARCH · Search NASA

Results for “data ingest”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

Goddard Conference on Mass Storage Systems and Technologies, volume 2

Papers and viewgraphs from the conference are presented. Discussion topics include the IEEE Mass Storage System Reference Model, data archiving standards, high-performance storage devices, magnetic and magneto-optic storage systems, magnetic and optical recording technologies, high-performance helical scan recording systems, and low end helical scan tape drives. Additional discussion topics addressed the evolution of the identifiable unit for processing (file, granule, data set, or some similar object) as data ingestion rates increase dramatically, and the present state of the art in mass storage technology.

Kobler, Ben↗

Telecommunications issues of intelligent database management for ground processing systems in the EOS era

Future NASA earth science missions, including the Earth Observing System (EOS), will be generating vast amounts of data that must be processed and stored at various locations around the world. Here we present a stepwise-refinement of the intelligent database management (IDM) of the distributed active archive center (DAAC - one of seven regionally-located EOSDIS archive sites) architecture, to showcase the telecommunications issues involved. We develop this architecture into a general overall design. We show that the current evolution of protocols is sufficient to support IDM at Gbps rates over large distances. We also show that network design can accommodate a flexible data ingestion storage pipeline and a user extraction and visualization engine, without interference between the two.

Touch, Joseph D.↗

Demonstrating Acquisition of Real-Time Thermal Data Over Fires Utilizing UAVs

A disaster mitigation demonstration, designed to integrate remote-piloted aerial platforms, a thermal infrared imaging payload, over-the-horizon (OTH) data telemetry and advanced image geo-rectification technologies was initiated in 2001. Project FiRE incorporates the use of a remotely piloted Uninhabited Aerial Vehicle (UAV), thermal imagery, and over-the-horizon satellite data telemetry to provide geo-corrected data over a controlled burn, to a fire management community in near real-time. The experiment demonstrated the use of a thermal multi-spectral scanner, integrated on a large payload capacity UAV, distributing data over-the-horizon via satellite communication telemetry equipment, and precision geo-rectification of the resultant data on the ground for data distribution to the Internet. The use of the UAV allowed remote-piloted flight (thereby reducing the potential for loss of human life during hazardous missions), and the ability to "finger and stare" over the fire for extended periods of time (beyond the capabilities of human-pilot endurance). Improved bit-rate capacity telemetry capabilities increased the amount, structure, and information content of the image data relayed to the ground. The integration of precision navigation instrumentation allowed improved accuracies in geo-rectification of the resultant imagery, easing data ingestion and overlay in a GIS framework. We focus on these technological advances and demonstrate how these emerging technologies can be readily integrated to support disaster mitigation and monitoring strategies regionally and nationally.

Ambrosia, Vincent G.↗

Effects of data selection on the assimilation of AIRS data

The Atmospheric InfraRed Sounder (AIRS), flying aboard NASA's Earth Observing System (EOS) Aqua satellite with the Advanced Microwave Sounding Unit-A (AMSU-A), has been providing data for use in numerical weather prediction (NWP) and data assimilation systems (DAS) for over three years. The full AIRS data set is currently not transmitted in near-real-time (NRT) to the NWP centers. Instead, data sets with reduced spatial and spectral information are produced and made available in NRT. In this paper, we evaluate the use of different channel selections and error specifications. We achieved significant positive impact from the Aqua AIRS/AMSU-A combination in both hemispheres during our experimental time period of January 2003. The best results were obtained using a set of 156 channels that did not include any in the 6.7micron water vapor band. The latter have a large influence on both temperature and humidity analyses. If observation and background errors are not properly specified, the partitioning of temperature and humidity information from these channels will not be correct, and this can lead to a degradation in forecast skill. We found that changing the specified channel errors had a significant effect on the amount of data that entered into the analysis as a result of quality control thresholds that are related to the errors. However, changing the channel errors within a relatively small window did not significantly impact forecast skill with the 155 channel set. We also examined the effects of different types of spatial data reduction on assimilated data sets and NWP forecast skill. Whether we picked the center or the warmest AIRS pixel in a 3x3 array affected the amount of data ingested by the analysis but had a negligible impact on the forecast skill.

Joiner, Joanna↗

Severe Weather and Weak Waterspout Checklist in MIDDS

The goal of this task was to migrate the functionality of the AMU web-based Severe Weather Forecast Decision Aid and the 45 WS Weak Waterspout Checklist to MIDDS, the operational data ingest and display system of the 45 WS. Forecasting the occurrence and timing of warm season severe weather and weak waterspouts is challenging for 45 WS operational personnel. These interactive tools assist forecasters in determining the probability of issuing severe weather watches and warnings for the day. MIDDS is able retrieve many of the needed parameter values for the worksheet automatically. The AMU was able to develop user-friendly tools in MIDDS for both of these tools using McBASI coded programs. The tools retrieve needed values from MIDDS automatically, and require the forecaster to answer a few subjective questions. Both tools were tested and previewed to the 45 WS on MIDDS. In their previous forms, the forecasters enter values into both tools manually to output a threat index. Making these tools more automatic will reduce the possibility of human error and increase efficiency.

Wheeler, Mark M.↗

Applied Meteorology Unit (AMU) Quarterly Report. First Quarter FY-05

This report summarizes the Applied Meteorology Unit (AMU) activities for the first quarter of Fiscal Year 2005 (October - December 2005). Tasks reviewed include: (1) Objective Lightning Probability Forecast: Phase I, (2) Severe Weather Forecast Decision Aid, (3) Hail Index, (4) Stable Low Cloud Evaluation, (5) Shuttle Ascent Camera Cloud Obstruction Forecast, (6) Range Standardization and Automation (RSA) and Legacy Wind Sensor Evaluation, (7) Advanced Regional Prediction System (ARPS) Optimization and Training Extension, and (8) User Control Interface for ARPS Data Analysis System (ADAS) Data Ingest

Bauman, William↗

Applied Meteorology Unit (AMU) Quarterly Report Fourth Quarter FY-04

This report summarizes the Applied Meteorology Unit (A MU) activities for the fourth quarter of Fiscal Year 2004 (July -Sept 2004). Tasks covered are: (1) Objective Lightning Probability Forecast: Phase I, (2) Severe Weather Forecast Decision Aid, (3) Hail Index, (4) Shuttle Ascent Camera Cloud Obstruction Forecast, (5) Advanced Regional Prediction System (ARPS) Optimization and Training Extension and (5) User Control Interface for ARPS Data Analysis System (ADAS) Data Ingest.

Bauman, William↗

ParaText : scalable text analysis and visualization.

Automated analysis of unstructured text documents (e.g., web pages, newswire articles, research publications, business reports) is a key capability for solving important problems in areas including decision making, risk assessment, social network analysis, intelligence analysis, scholarly research and others. However, as data sizes continue to grow in these areas, scalable processing, modeling, and semantic analysis of text collections becomes essential. In this paper, we present the ParaText text analysis engine, a distributed memory software framework for processing, modeling, and analyzing collections of unstructured text documents. Results on several document collections using hundreds of processors are presented to illustrate the exibility, extensibility, and scalability of the the entire process of text modeling from raw data ingestion to application analysis.

99 GENERAL AND MISCELLANEOUS↗

Visualizing Energy Resources Dynamically on Earth (VERDE)

VERDE is a software service that ingests data on real-time eneryg grid status and analyzes the data with models and algorithms presenting the output in a form that can be visualized by client spatio-temporal browsers

Source record↗

AGN-201 Digital Twin Concept of Operations

The AGN-201 is a nuclear reactor at Idaho State University (ISU) and is currently being used in the development of a digital twin (DT) with Idaho National Laboratory (INL). The goal is to create a DT (called the AGN-201 DT) which can monitor an operating nuclear reactor to determine when the reactor is being operated normally; normal operations are any operations that is declared by the ISU staff. To accomplish this, researchers at INL developed a DT ecosystem which can ingest data from the AGN-201 reactor. This data is fed into a series of machine learning (ML) and reactor physics models. The ML and reactor physics models assess the data and determine if the reactor is operating normally. Event(s) that are flagged as anomalies are investigated by the INL staff to determine if any undeclared experiments were conducted. The AGN-201 DT was verified in July/August 2023, when the ISU staff performed undeclared experiments and the INL staff were able to assess the anomalous events and determine what likely caused each event. This project was a steppingstone to promote the use of DTs for international safeguards and marks the first time a DT was able to monitor a nuclear reactor for this purpose. This document provides the concept of operations (CONOPS) for the AGN-201 digital twin (DT). The CONOPS describes how the AGN-201 and AGN-201 DT are expected to operate and details how each are established.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Machine Learning Digital Twin for Lithium Ion Battery State of Health Predictions

A digital twin system has been established to model the long term degradation of the state of health of lithium ion batteries. Two data streams result from the computational model of the system and from the physical experiment measurements. A machine learning pipeline has been developed at the nexus of these data streams. Leveraging the unique data sources in multiple transfer learning approaches has lead to the development of multiple cell specific machine learning digital twins. Our discussion on this first of its kind technology will cover challenges in data ingestion, scalability, architecture and orchestration, and research findings.

25 - ENERGY STORAGE↗

Optimizing Management of Persistent Data Structures in High-Performance Analytics

Large-scale data analytics workflows ingest massive input data into various data structures, including graphs and key-value datastores. These data structures undergo multiple transformations and computations and are typically reused in incremental and iterative analytics workflows. Persisting in-memory views of these data structures enables reusing them beyond the scope of a single program run while avoiding repetitive raw data ingestion overheads. Memory-mapped I/O enables persisting in-memory data structures without data serialization and deserialization overheads. However, memory-mapped I/O lacks the key feature of persisting consistent snapshots of these data structures for incremental ingestion and processing. The obstacles to efficient virtual memory snapshots using memory-mapped I/O include background writebacks outside the application’s control, and the significantly high storage footprint of such snapshots. To address these limitations, we present Privateer, a memory and storage management tool that enables storage-efficient virtual memory snapshotting while also optimizing snapshot I/O performance. Here, we integrated Privateer into Metall, a state-of-the-art persistent memory allocator for C++, and the Lightning Memory-Mapped Database (LMDB), a widely-used key-value datastore in data analytics and machine learning. Privateer optimized application performance by 1.22× when storing data structure snapshots to node-local storage, and up to 16.7× when storing snapshots to a parallel file system. Privateer also optimizes storage efficiency of incremental data structure snapshots by up to 11× using data deduplication and compression.

Computer science↗

Panorama 360 (Final Report)

This final technical report from the lead institution, USC grant #DE-SC0012636, serves as the final technical report for collaborative institution UNC-CH grant #DE-SC0012390. The goal was to develop a repository and associated capabilities for data collection, ingestion, and analysis for a broad class of DOE applications that span experimental and simulation science workflows. In particular, this work focuses on workflows that include experimental data generation at DOE facilities. The main activities of Panorama 360 include the development of: (1) A distributed repository that stores different types of workflow execution data (e.g., point and time series performance traces at fine- and coarse-grained levels); (2) A set of open-source data capture, curation, and publishing tools fully integrated with a state-of-the-art workflow management system that automates data ingestion to the repository and enables users to discover, query, and process data from the repository; (3) A set of analysis algorithms and machine learning based tools to perform analysis and characterization of the gathered data, which can be used to detect anomalous performance or system faults; and (4) Best practices and recommendations for workflow evaluation, analysis, execution, and architectures.

97 MATHEMATICS AND COMPUTING↗

AAM National Campaign Tech Talk: Data Pipeline Familiarization

NASA's AWS-based Data Pipeline allows real-time data submission and ingestion, with immediate monitoring of data rate, coverage and ingestion quality. Problems are immediately discovered and can be corrected with agility by both Partners and NASA during the a simulation or flight event​.

Data Pipeline↗

Data, Meet Compute: NASA's Cumulus Ingest Architecture

NASA's Earth Observing System Data and Information System (EOSDIS) houses nearly 30PBs of critical Earth Science data and with upcoming missions is expected to balloon to between 200PBs-300PBs over the next seven years. In addition to the massive increase in data collected, researchers and application developers want more and faster access - enabling complex visualizations, long time-series analysis, and cross dataset research without needing to copy and manage massive amounts of data locally. NASA has looked to the cloud to address these needs, building its Cumulus system to manage the ingest of diverse data in a wide variety of formats into the cloud. In this talk, we look at what Cumulus is from a high level and then take a deep dive into how it manages complexity and versioning associated with multiple AWS Lambda and ECS microservices communicating through AWS Step Functions across several disparate installations

EOSDIS↗

Characterizing the Relationships Among Lightning and Storm Parameters: Lightning as a Proxy Variable

We have gained important insights from prior studies that have suggested relationships between lightning and storm growth, decay, convective rain flux, vertical distribution of storm mass and echo volume in the region, and storm energetics. A study was initiated in the Summer of 1996 to determine how total (in-cloud plus ground) lightning observations might provide added knowledge to the forecaster in the determination and identification of severe thunderstorms and weather hazards in real-time. The Melbourne Weather Office was selected as a primary site to conduct this study because Melbourne is the only site in the world with continuous and open access to total lightning (LDAR) data and a Doppler (WSR-88D) radar. A Lightning Imaging Sensor Data Applications Demonstration (LISDAD) system was integrated into the forecaster's workstation during the Summer 1996 to allow the forecaster to interact in real-time with the multi-sensor data being displayed. LISDAD currently ingests LDAR data, the cloud-to-ground National Lightning Detection Network (NLDN) data, and the Melbourne radar data in f real-time. The interactive features provide the duty forecaster the ability to perform quick diagnostics on storm cells of interest. Upon selection of a storm cell, a pop-up box appears displaying the time-history of various storm parameters (e.g., maximum radar reflectivity, height of maximum reflectivity, echo-top height, NLDN and LDAR lightning flash rates, storm-based vertically integrated liquid water content). This product is archived to aid on detailed post-analysis.

Goodman, S. J.↗

NASA’s Atmospheric Science Data Center’s Approach to a Cloud-Based Model of Ingest, Archival, and Distribution of TEMPO Data: Methods, Challenges, and Best Practices

The National Aeronautics and Space Administration's (NASA) Atmospheric Science Data Center (ASDC) at NASA Langley Research Center in Hampton, VA provides atmospheric science data products and services to the science community, including enhanced search and subsetting capabilities for numerous datasets. The ASDC is the official Distributed Active Archive Center (DAAC) of record for the upcoming Tropospheric Emissions: Monitoring of Pollution (TEMPO) instrument. TEMPO will be situated on a geostationary satellite positioned at a longitude near the center of the conterminous United States and focused on North America, making hourly swaths of its field of regard from east to west. ASDC’s data products are currently hosted locally and services (e.g., spatial and temporal subsetting) are managed on premises. The ASDC is planning to provide TEMPO data and services in the cloud through the Earthdata Search platform. This presentation will discuss the ASDC’s approach to a cloud-based model of ingest, archival, and distribution of TEMPO data. Methods, challenges, best practices, lessons learned, and future plans will be discussed.

Iman Nasif↗

Collaborative Data Curation to Support the Multi-Mission Algorithm and Analysis Platform (MAAP)

Upcoming space-borne missions will offer unprecedented data about Earth but will also feature exponentially high data volumes. These high data volumes will change the way the scientific community works with data and will also create a unique need for improved data sharing and collaboration. NASA and ESA are working together to address these issues by collaboratively developing the Multi-Mission Algorithm and Analysis Platform (MAAP) to improve the understanding of global aboveground terrestrial carbon dynamics. The MAAP will support ESA’s BIOMASS mission, NASA’s GEDI mission and NASA/ISRO’s NISAR mission. The MAAP will be developed in two phases: a pilot phase and a full production phase. The pilot phase will demonstrate collaboration and basic capabilities. The pilot phase will focus on biomass relevant airborne and field campaign data. Two NASA teams are supporting the development of the MAAP. The MAAP engineering team is responsible for the development, maintenance and operations of the MAAP system while the MAAP data team ensures the ongoing quality of the data, metadata and other information provided in the MAAP. The MAAP data team also supports the ingest and archive of identified data to the MAAP platform. This poster describes the use case development process for the pilot MAAP and the data curated in support of those use cases. Additionally, this presentation will outline the pilot MAAP data ingest process and metadata curation effort along with efforts to ensure interoperability between ESA and NASA data and metadata.

Bugbee, Kaylin↗