Search NASA⌕ Search

SEARCH · Search NASA

Results for “data ingest”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Plenario: An Open Data Discovery and Exploration Platform for Urban Science

The past decade has seen the widespread release of open data concerning city services, conditions, and activities by government bodies and public institutions of all sizes. Hundreds of open data portals now host thousands of datasets of many different types. These new data sources represent enormous po- tential for improved understanding of urban dynamics and processes—and, ultimately, for more livable, efficient, and prosperous communities. However, those who seek to realize this potential quickly discover that discovering and applying those data relevant to any particular question can be extraordinarily dif- ficult, due to decentralized storage, heterogeneous formats, and poor documentation. In this context, we introduce Plenario, a platform designed to automating time-consuming tasks associated with the discovery, exploration, and application of open city data—and, in so doing, reduce barriers to data use for researchers, policymakers, service providers, journalists, and members of the general public. Key innovations include a geospatial data warehouse that allows data from many sources to be registered into a common spatial and temporal frame; simple and intuitive interfaces that permit rapid discovery and exploration of data subsets pertaining to a particular area and time, regardless of type and source; easy export of such data subsets for further analysis; a user-configurable data ingest framework for automated importing and periodic updating of new datasets into the data warehouse; cloud hosting for elastic scaling and rapid creation of new Plenario instances; and an open source implementation to enable community contributions. We describe here the architecture and implementation of the Plenario platform, discuss lessons learned from its use by several communities, and outline plans for future work.

Catlett, Charlie↗

A Nonrelational Data Warehouse for the Analysis of Field and Laboratory Data From Multiple Heterogeneous Photovoltaic Test Sites

A nonrelational, distributed computing, data warehouse, and analytics environment (Energy-CRADLE) was developed for the analysis of field and laboratory data from multiple heterogeneous photovoltaic (PV) test sites. This data informatics and analytics infrastructure was designed to process diverse formats of PV performance data and climatic telemetry time-series data collected from a PV outdoor test network, i.e., the Solar Durability and Lifetime Extension global SunFarm network, as well as point-in-time laboratory spectral and image measurements of PV material samples. Using Hadoop/HBase for the distributed data warehouse, Energy-CRADLE does not have a predefined data table schema, which enables ingestion of data in diverse and changing formats. For easy data ingestion and data retrieval, Energy-CRADLE utilizes Hadoop streaming to enable Python MapReduce and provides a graphical user interface, i.e., py-CRADLE. By developing the Hadoop distributed computing platform and the HBase NoSQL database schema for solar energy, Energy-CRADLE exemplifies an integrated, scalable, secure, and user-friendly data informatics and analytics system for PV researchers. Furthermore, an example of Energy-CRADLE enabled scalable, data-driven, analytics is presented, where machine learning is used for anomaly detection across 2.2 million real-world current-voltage (I–V) curves of PV modules in three distinct Koppen–Geiger climatic zones.

14 SOLAR ENERGY↗

White Paper: Scalable Digital Twin Capabilities for Aging and Surveillance of Engineered Systems

This white paper presents a multi-year initiative to develop practical, secure, and scalable digital twin capabilities for engineered systems in aging and surveillance contexts—an approach pioneered at the National Nuclear Security Administration (NNSA) Lawrence Livermore National Laboratory (LLNL) that maps directly onto the needs and ambitions of the Navy for ship- and fleet-level digital twins. LLNL’s work in building part- and process-level digital twins for advanced manufacturing, with a vision to scale up to entire factory floors and, ultimately, enterprise-wide digital twins, offers an adaptable pathway for the Navy as it seeks to modernize lifecycle management, readiness, and predictive maintenance across ships and fleets. For our application, we integrate physics-based modeling with automated data ingestion, processing, and AI-driven calibration, creating hybrid models that are both interpretable and data responsive. We modernized legacy workflows, established centralized data infrastructure, automated experimental pipelines, and demonstrated end-to-end coupling of accelerated aging data with finite element simulations via optimization and surrogate modeling. The result is a generalizable framework that supports part-level digital twins today and lays the groundwork for future system-level twins suitable for Navy applications.

36 MATERIALS SCIENCE↗

Cumulus: NASA's Cloud Based Distributed Active Archive Center Prototype

NASAs Earth Science Data System (ESDS) Program serves as a central cog in order to facilitate the implementation of NASA's Earth Science strategic plan. Since 1994, the ESDS Program has committed to the full and open sharing of Earth science data obtained from NASA instruments to all users. One of the key responsibilities of the ESDS Program is to continuously evolve the entire data and information system to maximize returns on the collected NASA data. An independent review was conducted in 2015 to holistically review the EOSDIS in order to identify gaps. The review recommendations were to investigate two areas: one, whether commercial cloud providers offer potential for storage, processing, and operational efficiencies, and two, the potential development of new data access and analysis paradigms. In response, ESDS has initiated several prototypes investigating the advantages and risks of leveraging cloud computing. This paper describes one such prototyping activity named Cumulus. Cumulus is being designed and developed as a "native" cloud-based data ingest, archive and management system that can be used for all future NASA Earth science data streams. Cumulus will foster design of new analysisvisualization tools that can leverage collocated data from all of the distributed DAACs as well as elastic cloud computing resources.

Earth Science Informatics↗

Radarsat Processing System at ASF

Radarsat is a Canadian polar orbiting remote sensing satellite scheduled for launch in September 1995. Its lone instrument on-board is a synthetic aperture radar (SAR) that is capable of operating in a number of imaging modes including the first operational ScanSAR mode. As one of the data reception, processing and archive facility for Radarsat data, Alaska SAR Facility (ASF) has responded to its Science users by establishing a Radarsat processing system to handle the data processing of all Radarsat modes. This task involves enhancements to the high throughput hardware based Alaska SAR Processor (ASP) to handle standard mode Radarsat data in addition to its existing ERS and JERS capabilities, the addition of the new ScanSAR Processor (SSP) to process the Radarsat ScanSAR mode data, and the introduction of a Precision Processor (PP) to accommodate the special Radarsat modes such as fine resolution and wide swath. For raw data ingestion and distribution to the appropriate SAR processor, a new Control Processor (CP) and Raw Data Scanner (RDS) subsystem is also incorporated.

Radarsat↗

Testing Mars Exploration Rover-Inspired Operational Strategies for Semi-Autonomous Rovers on the Moon Ii: the GeoHeuristic Operational Strategies Test in Alaska

We used MER-derived semi-autonomous rover science operations strategies to determine best practices suitable for remote semi-autonomous lunar rover geology. Two field teams studied two glacial moraines as analogs for potential ice-bearing lunar regolith. At each site a Rover Team commanded a human rover to execute observations based on common MER sequences; the resulting data were used to identify and characterize targets of interest. A Tiger Team followed the Rover Team using traditional terrestrial field methods, and the results of the two teams were compared. Narrowly defined goals that can be addressed using cm-scale or coarser resolution may be met sufficiently by the operational strategies adapted from MER survey mode. When reconnaissance is the primary goal, the strategies tested are necessary but not sufficient. Further, there may be a set of optimal observations for such narrowly defined, hypothesis-driven science goals, such that collecting further data would result in diminishing returns. We confirm results of previous tests that indicated systematic observations might improve efficiency during strategic planning, and improve science output during data analysis. This strategy does not markedly improve the rate at which a science team can ingest data to feed back into tactical decision-making. Other methods should be tested to separate the strategic and tactical processes, and to build in time for data analysis.

MER↗

NASA Open Science Data Repository: Maximizing Spaceflight Bioscience Data

The next era in human space exploration is rapidly approaching and will require the use of countermeasures to deep space health hazards. The development of countermeasures (or, the re-purposing of existing agents) will be highly dependent on our understanding of basic biological responses to space stressors (e.g. ionizing radiation, altered gravitational fields, altered day-night cycles, confinement, isolation, hostile-closed environments, distance-duration from Earth, exposure to celestial regolith, etc.). The fast-growing array of space biological data, which in the past was simply archived after minimal analysis, holds great potential if it can be reorganized and formatted for data re-analysis and re-use via Open Science. Organizing the data for such analysis is a challenge because of its diverse nature (molecular, cellular, tissue, imaging, whole organism and behavior). To address the challenges posed by gaining new knowledge from a vast and diverse amount of biological, health and environmental data in space, the NASA Open Science Data Repository (OSDR - osdr.nasa.gov/bio) plays a crucial role in curating and openly publishing biological data from space-related experiments. Its design incorporates successes and lessons from NASA GeneLab, encompassing not only high-throughput sequencing data but also physiological, phenotypic, and telemetry data. The OSDR makes space biological data FAIR (findable, accessible, interoperable, reusable), and facilitates effective data ingestion, dissemination, and Open Science collaborations. The OSDR also has the capability to integrate human astronaut data with state-of-the-art security and accessibility procedures. We will discuss here several strategies that NASA’s Biological and Physical Science Division have put in place to maximize the return on investment for spaceflight bioscience data.

space biology↗

Utilizing the Dynamic Networks Data Processing and Analysis Experiment (DNE18) to Establish Methodologies for the Comparison of Automatic Infrasonic Signal Detectors

The Dynamic Networks Experiment 2018 (DNE18) was a collaborative effort between Los Alamos National Laboratory (LANL), Sandia National Laboratories (SNL), Lawrence Livermore National Laboratory (LLNL) and Pacific Northwest National Laboratory (PNNL) designed to evaluate methodologies for multi-modal data ingestion and processing. One component of this virtual experiment was a quantitative assessment of current capabilities for infrasound data processing, beginning with the establishment of a baseline for infrasound signal detection. To produce such baselines, SNL and LANL exploited a common dataset of infrasound data recorded across a regional network in Utah from December 2010 through February 2011. We utilize two automated signal detectors, the Adaptive F-Detector (AFD) and the Multivariate Adaptive Learning Detector (MALD) to produce automated signal detection catalogs and an analyst-produced catalog. Comparisons indicate that automatic detectors may be able to identify small amplitude, low SNR events that cannot be identified by analyst review. We document detector performance in terms of precision and recall, demonstrating that the AFD is more precise, but the MALD has higher recall. We use a synthetic dataset of signals embedded in pink noise in order to highlight shortcomings in assessing detection algorithms for low signal to noise ratio signals which are commonly of interest to the nuclear monitoring community. For comparisons utilizing the synthetic dataset, the AFD has higher recall while precision is equal for both detectors. These results indicate that both detectors perform well across a variety of background noise environments; however, both detectors fail to identify repetitive, short duration signals arriving from similar backazimuths. These failures represent specific scenarios that could be targeted for further detector development.

97 MATHEMATICS AND COMPUTING↗

Automated Cloud Based Long Short-Term Memory Neural Network Based SWE Prediction

Snow derived water is a critical component of the US water supply. Measurements of the Snow Water Equivalent (SWE) and associated predictions of peak SWE and snowmelt onset are essential inputs for water management efforts. This paper aims to develop an integrated framework for real-time data ingestion, estimation, prediction and visualization of SWE based on daily snow datasets. In particular, we develop a data-driven approach for estimating and predicting SWE dynamics using the Long Short-Term Memory neural network (LSTM) method. Our approach uses historical datasets (precipitation, air temperature, SWE, and snow thickness) collected at NRCS Snow Telemetry (SNOTEL) stations to train the LSTM network and current year data to predict SWE behavior. The performance of our prediction was compared for different prediction dates and prediction training datasets. Our results suggest that the proposed LSTM network can be an efficient tool for forecasting the SWE timeseries, as well as Peak SWE and snowmelt timing. Results showed that the window size impacts the model performance (where the Nash Sutcliffe efficiency (NSE) ranged from 0.96 to 0.85 and the Rooted Mean Square Error (RMSE) ranged from 0.038 to 0.07) with an optimum number that should be calibrated for different stations and climate conditions. In addition, by implementing the LSTM prediction capability in a cloud based site-monitoring platform, we automate model-data integration. By making the data accessible through a graphical web interface and an underlying API which exposes both training and prediction capabilities. The associated results can be made easily accessible to a broad range of stakeholders.

54 ENVIRONMENTAL SCIENCES↗

Exploring for Superhot Geothermal Targets in Magmatic Settings: 2022 Field Campaign at Newberry Volcano

This paper presents preliminary results from a subset of work carried out as part of a multinational research project entitled DErisking Exploration for multiple geothermal Plays in magmatic ENvironments (DEEPEN), supported by the U.S. Department of Energy (DOE) and Geothermica, a joint effort by EU member states and associated countries. The DEEPEN project will develop a customized approach to exploration for supercritical and superhot geothermal plays in magmatic systems, which will be applied to two demonstration sites. This paper summarizes field activities carried out at the U.S. demonstration site, Newberry Volcano in central Oregon. The objective of this work effort is to refine the subsurface model of Newberry Volcano, with special focus on deeper zones including the magmatic plumbing system and other key geologic elements. New data collection included gravity and wideband magnetotelluric (MT) surveys, as well as reinstallation of a seismic network. The National Renewable Energy Laboratory (NREL) and Enthalpion Energy LLC (Enthalpion) worked with the Deschutes National Forest Fort Rock District to use a low ground disturbance method of MT deployment to collect MT data inside the caldera and other restricted areas inside the National Volcanic Monument. This opened these areas to geophysical exploration for the first time in decades. Sites along and adjacent to the south rim of the volcano constituted the primary survey objectives. A team from Lawrence Berkeley National Laboratory (LBNL), the U.S. Geological Survey (USGS), and AltaRock also began the process of reinstalling the seismic network from the AltaRock enhanced geothermal system (EGS) demonstration in anticipation of further development activities at the site. The data ingestion, reduction, and analysis phase of the project is ongoing. We are currently processing the MT and gravity data and are developing a new, highly GPU-accelerated, 3D joint MT and gravity inversion to better localize the south rim/south flank conductive target and better understand its relationship to deep heat, fluid sources, and surface extrusive features. Joint inversions, which have not yet been undertaken at Newberry, will allow us to obtain constraints on the geologic model that cannot be determined from each method in isolation, improving our ability to image key geologic features at depth.

geophysics↗

High rate science data handling on Space Station Freedom

A study by NASA's User Information System Working Group for Space Station Freedom (SSF) has determined that the proposed onboard Data Management System, as initially configured, will be incapable of handling the data-generation rates typical of numerous scientific sensor payloads; many of these generate data at rates in excess of 10 Mbps, and there are at least four cases of rates in excess of 300 Mbps. The SSF Working Group has accordingly suggested an alternative conceptual architecture based on technology expected to achieve space-qualified status by 1995. The architecture encompasses recorders with rapid data-ingest capabilities and massive storage capabilities, optical delay lines allowing the recording of only the phenomena of interest, and data flow-compressing image processors.

Handley, Thomas H., Jr.↗

Super Resolution for Renewable Energy Resource Data With Wind From Reanalysis Data (Sup3rWind) and Application to Ukraine [Slides]

In this work we present a novel deep learning-based downscaling method, using generative adversarial networks (GANs), for generating high-resolution wind resource data from ECMWF Reanalysis v5 data (ERA5). We show that by training a GAN model on ERA5, as opposed to coarsened high-resolution data, we achieve results that are competitive with conventional dynamical downscaling. This GAN-based downscaling method additionally reduces computational costs over dynamical downscaling by two orders of magnitude. All GANs are trained on data sampled from CONUS, selected to provide a diverse sampling of terrain conditions, and validated on observational data along with data held out from training. This cross-validation shows low error and high correlations with observations and excellent agreement with hold out data across physical distributions. Our approach is finally used to downscale 30km hourly ERA5 to 2-km 5-minute wind data, for January 2000 through December 2023, at multiple hub heights, over Ukraine, Moldova, and part of Romania. Comparisons against observational data from Meteorological Assimilation Data Ingest System (MADIS) and multiple wind farms show the same level of performance as for CONUS validation. This 24 year data record is the first member of the "super resolution for renewable energy resource data with wind from reanalysis data" dataset (Sup3rWind).

17 WIND ENERGY↗

EXSCLAIM!

Due to recent improvements in image resolution and acquisition speed, materials microscopy is experiencing an explosion of published imaging data. The standard publication format, while sufficient for traditional data ingestion scenarios where a select number of images can be critically examined and curated manually, is not conducive tolarge-scale data aggregation or analysis, hindering data sharing and reuse. Most images in publications are presented as components of a larger figure with their explicit context buried in the main body or caption text, so even if aggregated, collections of images with weak or no digitized contextual labels have limited value. To solve the problem of curating labeled microscopy data from literature, we introduce the EXSCLAIM! Python toolkit for the automatic EXtraction, Separation, and Caption-based natural Language Annotation of IMages from scientific literature. The software is implemented through a three part pipeline: the JournalScraper, which searches the web and downloads figures and captions based on a user provided query, the CaptionDistributor, which separates caption text based on the subfigure each portion of the caption refers to, and the FigueSeparator, which separates figures into component subfigures and extracts other visual information. Also included is a Django user interface for exploring the resulting dataset.

CHAN, MARIA↗

ParaText : scalable text modeling and analysis.

Automated processing, modeling, and analysis of unstructured text (news documents, web content, journal articles, etc.) is a key task in many data analysis and decision making applications. As data sizes grow, scalability is essential for deep analysis. In many cases, documents are modeled as term or feature vectors and latent semantic analysis (LSA) is used to model latent, or hidden, relationships between documents and terms appearing in those documents. LSA supplies conceptual organization and analysis of document collections by modeling high-dimension feature vectors in many fewer dimensions. While past work on the scalability of LSA modeling has focused on the SVD, the goal of our work is to investigate the use of distributed memory architectures for the entire text analysis process, from data ingestion to semantic modeling and analysis. ParaText is a set of software components for distributed processing, modeling, and analysis of unstructured text. The ParaText source code is available under a BSD license, as an integral part of the Titan toolkit. ParaText components are chained-together into data-parallel pipelines that are replicated across processes on distributed-memory architectures. Individual components can be replaced or rewired to explore different computational strategies and implement new functionality. ParaText functionality can be embedded in applications on any platform using the native C++ API, Python, or Java. The ParaText MPI Process provides a 'generic' text analysis pipeline in a command-line executable that can be used for many serial and parallel analysis tasks. ParaText can also be deployed as a web service accessible via a RESTful (HTTP) API. In the web service configuration, any client can access the functionality provided by ParaText using commodity protocols ... from standard web browsers to custom clients written in any language.

99 GENERAL AND MISCELLANEOUS↗

Code Artifact for: Clustering Analysis of Commercial Vehicles Using Automatically Extracted Features from Time Series Data [SWR-21-96]

This repository contains data ingestion, feature extraction, and analysis code used in NREL Technical report "Clustering Analysis of Commercial Vehicles Using Automatically Extracted Features from Time Series Data." The code is written in Python. The ETL and feature extraction code must be run in a Spark context. The analysis code can be run without Spark, provided you have pre-computed features in a CSV file. Analysis code related to the NREL Technical Report NREL/TP-2C00-74212. Includes PySpark functions to perform trip segmentation and feature extraction over big time series data in Apache Spark. Includes "domain specific" features such as Aerodynamic Speed (ft/s), Characteristic Acceleration (ft/s2), Percent Below 55 (%), Percent Zero (%), Stops Per Mile, Average Speed (mph), Maximum Speed (mph), and Speed Standard Deviation (mph). Includes Pyspark UDF to compute "domain agnostic" features using the TSFresh library. This software record also includes the analysis notebooks and code to generate the results in the previously mentioned technical report.

Perr-Sauer, Jordan↗

An Approach to Data Management Planning for Protected Data Projects

This report describes what is required in a data management plan for data that needs to be protected in some fashion. Here we provide an overview of what a data manager should consider, including the data ingestion and various extract-transform-load processes, metadata considerations and documentation, to what might need to be accounted for in the event of data loss. In addition, this report includes two appendices: forms that, when filled out, make the user compliant with DOE data management requirements as well as additional requirements for handling protected data at ORNL.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Impact of Assimilating AIRS Cloud-Cleared Radiances on Atmospheric Dynamics and Boundary Layer Height at High Latitudes

High latitude weather forecasts, on scales ranging from mesoscale to synoptic, present difficulties due, in part, to the sparsity of conventional observations. In addition, the prevalence of extended low-level stratus cloud cover limits the use of infrared data, which are operationally assimilated only in areas unaffected by clouds. Use of cloud-cleared AIRS (Atmospheric Infrared Sounder) radiances (AIRS CCR), allows the assimilation of infrared information in cloudy regions, permitting data ingestion in regions usually undersampled. This study explores the sensitivity of planetary boundary layer height and related atmospheric dynamics to the assimilation of these data in the Goddard Earth Observing System (GEOS, version 5) data assimilation and forecast system during the boreal fall 2014 season using observing system experiments (OSEs). Examined here are comparisons between the current, operational approach of assimilating AIRS clear-sky radiances against the assimilation of CCR. Assimilation of hyperspectral infrared information from AIRS over the Arctic region slightly modifies the lower midtropospheric temperature structure, which in turn contributes to adjustments in geopotential height, affecting the baroclinic instability properties over the entire hemisphere and explaining the overall improvement in global forecast skill.

McGrath-Spangler, E. L.↗

Application of TOMS data to weather analysis models

Recent studies of the errors of analysis reflected by numerical weather analysis models are reviewed. Despite the improvements in data coverage and data ingestion methods in the past decade, it is found that significant errors of analysis persist. A case study comparison of Total Ozone Mapping Spectrometer (TOMS) data and aircraft data with the NMC analysis over the North Atlantic is used to illustrate the size of local errors encountered. The possibility of using TOMS images to locate meteorologically significant features such as troughs and ridges near the tropopause is discussed.

Nastrom, Gregory D.↗