Search NASA⌕ Search

SEARCH · Search NASA

Results for “data ingest”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16

Performance Analysis Tool for HPC and Big Data Applications on Scientific Clusters

Big data is prevalent in HPC computing. Many HPC projects rely on complex workflows to analyze terabytes or petabytes of data. These workflows often require running over thousands of CPU cores and performing simultaneous data accesses, data movements, and computation. It is challenging to analyze the performance involving terabytes or petabytes of workflow data or measurement data of the executions, from complex workflows over a large number of nodes and multiple parallel task executions. To help identify performance bottlenecks or debug the performance issues in large-scale scientific applications and scientific clusters, we have developed a performance analysis framework, using state-ofthe- art open-source big data processing tools. Our tool can ingest system logs and application performance measurements to extract key performance features, and apply the most sophisticated statistical tools and data mining methods on the performance data. It utilizes an efficient data processing engine to allow users to interactively analyze a large amount of different types of logs and measurements. To illustrate the functionality of the big data analysis framework, we conduct case studies on the workflows from an astronomy project known as the Palomar Transient Factory (PTF) and the job logs from the genome analysis scientific cluster. Our study processed many terabytes of system logs and application performance measurements collected on the HPC systems at NERSC. The implementation of our tool is generic enough to be used for analyzing the performance of other HPC systems and Big Data workows.

97 MATHEMATICS AND COMPUTING↗

Enabling Space Biological Knowledge Discovery Through Image and Video Data Sharing

Increased biomedical risks associated with deep space crewed missions (cis-Lunar, Mars transit/surface) require development of health countermeasures, novel ecosystem support, risk modeling, and fundamental space biological knowledge discovery. Molecular-omics, physiological-phenotypic-behavioral, and environmental-radiation telemetry data from space biological and health studies are needed for reuse by scientists to address these tasks. The data as well as space-relevant biospecimens are being made more findable, accessible, interoperable, and reusable through NASA’s Open Science Data Repository (OSDR). This new OSDR umbrella grouping includes NASA GeneLab, the NASA Ames Life Sciences Data Archive (ALSDA), and the NASA Biological Institutional Scientific Collection. The OSDR system design appropriately handles metadata and processed-tabular results from ALSDA studies collected from space experiments. But raw and processed ALSDA bioimage and video datasets require an expansion of OSDR’s data architecture to handle ingestion, curation, and egress. The academic-industry bioimaging field saw a scientific renaissance in the past several years through leveraging open-source software, international collaborations, machine learning, and other open science/programming approaches. As crewed missions and more biological experiments are on the deep space horizon, OSDR is embracing data stewardship through listening to feedback from subject matter experts and designing an expanded architecture which is appropriate for NASA’s goals to enable analysis and reuse of bioimaging and video data for the public science community.Discovery Through Image and Video Data Sharing

space biology↗

A Unified Level of Service Model for NASA Earth Science Data Stewardship

During the past year, the Interagency Implementation and Concepts Team (IMPACT) reviewed existing service models in use at various NASA data centers in an effort to produce a unified, cohesive, and comprehensive Level of Service Model for all of NASA Distributed Active Archive Centers (DAACs). NASA DAACs are responsible for ensuring NASA Earth Science data are accurately and securely ingested, distributed, supported, and preserved. The term “Service" as used here refers to the spectrum of data management activities and outputs provided by DAACs in support of the data cared fo by each data cente. The unified Level-of- Service (LoS) model described in this presentation utilizes both the NASA-defined data product category and the data processing level to easily identify an appropriate level-of-service to be applied to a data product throughout the full data life cycle. This LoS model is to be used by DAACs when appraising incoming data in order to determine the appropriate and required services to provide. The LoS model utilizes a 3-level system in which services build upon previous levels and thereby require greater commitment and effort both on the part of the DAAC personnel and the data producer at the highest level. The LoS model description also contains examples of ways to communicate with data producers and data users what services can be expected, thereby bringing more consistent user experiences across the enterprise. In this presentation, we will outline the features of the LoS model and describe how it relates to the FAIR data practices and the NOAA Maturity Matrix model.

Smith, Deborah↗

Version 3 of the SMAP Level 4 Soil Moisture Product

The NASA Soil Moisture Active Passive (SMAP) Level 4 Soil Moisture (L4_SM) product provides 3-hourly, 9-km resolution, global estimates of surface (0-5 cm) and root zone (0-100 cm) soil moisture as well as related land surface states and fluxes from 31 March 2015 to present with a latency of 2.5 days. The ensemble-based L4_SM algorithm is a variant of the Goddard Earth Observing System version 5 (GEOS-5) land data assimilation system and ingests SMAP L-band (1.4 GHz) Level 1 brightness temperature observations into the Catchment land surface model. The soil moisture analysis is non-local (spatially distributed), performs downscaling from the 36-km resolution of the observations to that of the model, and respects the relative uncertainties of the modeled and observed brightness temperatures. Prior to assimilation, a climatological rescaling is applied to the assimilated brightness temperatures using a 6 year record of SMOS observations. A new feature in Version 3 of the L4_SM data product is the use of 2 years of SMAP observations for rescaling where SMOS observations are not available because of radio frequency interference, which expands the impact of SMAP observations on the L4_SM estimates into large regions of northern Africa and Asia. This presentation investigates the performance and data assimilation diagnostics of the Version 3 L4_SM data product. The L4_SM soil moisture estimates meet the 0.04 m3m3 (unbiased) RMSE requirement. We further demonstrate that there is little bias in the soil moisture analysis. Finally, we illustrate where the assimilation system overestimates or underestimates the actual errors in the system.

GEOS-↗

Lightning Mapping and the Nowcasting of Severe Storms

This paper describes a successful research and operational collaboration between NASA scientists and NWS forecasters to improve severe stor m warnings using real-time data from a regional VHF total lightning mapping array (LMA). Key objectives of our research using LMA data ar e: a) Identification of intensifying and weakening storms using the time rate-of-change of total flash rate; b) Improved severe storm poten tial situational awareness; c) Evaluation of the potential of total f lash rate trend to improve severe storm probability of detection (POD ) and lead time; and d) Validation of mesoscale model forecasts of th understorm initiation. The LMA data are distributed for ingest and di splay in the WFO AWIPS decision support system, and archived at each WFO for case studies, event playbacks, and assessments using the NWS Warning Event Simulator. The Huntsville WFO has upgraded severe thund erstorm warnings to verified tornado warnings and avoided a false ala rm on a severe storm through the added information on storm growth, intensification, and decay that can be deduced from the magnitude and temporal trend of total flash rates. We present detailed case studies of the observed relationships between lightning activity and tornadi c storm development as determined by radar reflectivity and velocity fields, and thunderstorms forecast by the Weather Research and Foreca st (WRF) model. From these collaborative studies, forecasters can eva luate the added value of total lightning data within the forecast and warning decision-making process (http://weather.msfc.nasa.gov/sport) .

Goodman, S.↗

Data Product: Ka ARM Zenith Radar (KAZR), in CfRadial format: general mode, corrected

The ‘kazrcfrcorge’ dataset is created by the KAZRCFRCOR VAP, which operates on ingested KAZR cfRadial-formatted data in conjunction with interpolated soundings to create significant detection masks, correct KAZR reflectivities for gaseous attenuation, and dealias mean Doppler velocities. This dataset serves as an input to the KAZR-ARSCL VAP.

54 ENVIRONMENTAL SCIENCES↗

Lessons Learned While Exploring Cloud-Native Architectures for NASA EOSDIS Applications and Systems

NASA's Earth Observing System (EOS) is a coordinated series of satellites for long term global observations. NASA's Earth Observing System Data and Information System (EOSDIS) is a multi-petabyte-scale archive of environmental data that supports global climate change research by providing end-to-end services from EOS instrument data collection to science data processing to full access to EOS and other earth science data. On a daily basis, the EOSDIS ingests, processes, archives and distributes over 3 terabytes of data from NASA's Earth Science missions representing over 6000 data products ranging from various types of science disciplines. EOSDIS has continually evolved to improve the discoverability, accessibility, and usability of high-impact NASA data spanning the multi-petabyte-scale archive of Earth science data products. Reviewed and approved by Chris Lynnes.

Cumulus↗

Inhalation and Ingestion Intakes with Associated Dose Estimates for Level II and Level III Personnel Using Capstone Study Data

Depleted uranium (DU) intake rates and subsequent dose rates were estimated for personnel entering armored combat vehicles perforated with DU penetrators (level II and level III personnel) using data generated during the Capstone Depleted Uranium (DU) Aerosol Study. Inhalation intake rates and associated dose rates were estimated from cascade impactors worn by sample recovery personnel and from cascade impactors that served as area monitors. Ingestion intake rates and associated dose rates were estimated from cotton gloves worn by sample recovery personnel and from wipe test samples from the interior of vehicles perforated with large caliber DU munitions. The mean DU inhalation intake rate for level II personnel ranged from 0.447 mg h-1 based on breathing zone monitor data (in and around a perforated vehicle) to 14.5 mg h-1 based on area monitor data (in a perforated vehicle). The mean DU ingestion intake rate for level II ranged from 4.8 mg h-1 to 38.9 mg h-1 based on the wipe test data including surface to glove transfer factors derived from the Capstone data. Based on glove contamination data, the mean DU ingestion intake rates for level II and level III personnel were 10.6 mg h-1 was and 1.78 mg h-1, respectively. Effective dose rates and peak kidney uranium concentration rates were calculated based on the intake rates. The peak kidney uranium concentration rate cannot be multiplied by the total exposure duration when multiple intakes occur because uranium will clear from the kidney between the exposures.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

A National Infrastructure for Artificial Intelligence on the Grid (NI4AI) (Final Scientific/Technical Report)

Electric utilities have traditionally taken a very pragmatic yet myopic approach with grid sensors and the resulting collected data. Sensors are purchased and deployed to solve a specific, known problem that has risen to sufficient awareness as to justify the effort of deploying sensors and the needed capital investment. This sensor data flows into proprietary software packages with limited functionality intended only to address the initial problem. This approach aligns with the financial incentives of the utility to deploy capital into fixed hardware assets for which the corporations earn a rate of return. This mentality stands in stark contrast to the big data revolution that started nearly 25 years ago with the rise of Google. In this worldview, data is a fundamental business asset; successful organizations collect, store, explore, merge, and exploit as much data as possible to not only solve problems well understood today but also to tackle new problems that will inevitably rise tomorrow. The ARPA-E Open Innovation 2018 project entitled A National Infrastructure for Artificial Intelligence on the Grid or NI4AI for short was designed to demonstrate this alternative paradigm for using data. To do this, the project was composed of three key thrust areas. The first major component deployed a variety of high-frequency grid sensors and captured terabytes of both wide-scale and localized grid measurements, generating high-value datasets for grid research and algorithm development. The second aspect made available PingThings’ PredictiveGridTM, a horizontally scalable, cloud-based data management and AI platform built for time series data to explore and exploit the collected data. Finally, the project fostered a diverse and open research community composed of experts from numerous fields through focused educational content, code sharing, and data science competitions. Shifting away from “single use” sensors and closed data silos within electric utilities is a major benefit to the public at large. This legacy approach to data is incredibly (1) capital intensive (new sensors must be deployed for each new problem and problems tend to arise continuously) and (2) painfully slow (new problems must be identified first and then new sensors must be deployed to collect data to begin to address the issue). The transition to a carbon neutral grid requires a massive transformation of the existing grid infrastructure and will continue to challenge the legacy grid in unforeseen ways. The only way to make the energy transition cost effective is for utilities to abandon this dated data paradigm and adopt more contemporary approaches. NI4AI has shown that it is technically possible and economically feasible to ingest, explore, and exploit grid data collected from even very high frequency sensing, such as continuous point on wave sensors collecting measurements 10,000 times a second. In fact, the PredictiveGrid platform used is commercially available and deployed at several utilities in the United States. Project accomplishments were numerous and included (1) making available a state of the art time series platform to the community, (2) collecting over 520 streams of time series data from grid sensors totaling over 1 trillion grid measurements, and (3) developing and nurturing a community within the industry focused on the use of data to create value for utilities and, ultimately, end consumers.

97 MATHEMATICS AND COMPUTING↗

The mRNA expression and histological integrity in rat forebrain motor and sensory regions are minimally affected by acrylamide exposure through drinking water

A study was undertaken to determine whether alterations in the gene expression or overt histological signs of neurotoxicity in selected regions of the forebrain might occur from acrylamide exposure via drinking water. Gene expression at the mRNA level was evaluated by cDNA array and/or RT-PCR analysis in the striatum, substantia nigra and parietal cortex of rat after a 2-week acrylamide exposure. The highest dose tested (maximally tolerated) of approximately 44 mg/kg/day resulted in a significant decreased body weight, sluggishness, and locomotor activity reduction. These physiological effects were not accompanied by prominent changes in gene expression in the forebrain. All the expression changes seen in the 1200 genes that were evaluated in the three brain regions were <= 1.5-fold, and most not significant. Very few, if any, statistically significant changes were seen in mRNA levels of the more than 50 genes directly related to the cholinergic, noradrenergic, GABAergic or glutamatergic neurotransmitter systems in the striatum, substantia nigra or parietal cortex. All the expression changes observed in genes related to dopaminergic function were less than 1.5-fold and not statistically significant and the 5HT1b receptor was the only serotonin-related gene affected. Therefore, gene expression changes were few and modest in basal ganglia and sensory cortex at a time when the behavioral manifestations of acrylamide toxicity had become prominent. No histological evidence of axonal, dendritic or neuronal cell body damage was found in the forebrain due to the acrylamide exposure. As well, microglial activation was not present. These findings are consistent with the absence of expression changes in genes related to changes in neuroinflammation or neurotoxicity. Over all, these data suggest that oral ingestion of acrylamide in drinking water or food, even at maximally tolerable levels, induced neither marked changes in gene expression nor neurotoxicity in the motor and somatosensory areas of the central nervous system.

60 APPLIED LIFE SCIENCES↗

Real-time Multi-granular Analytics Framework for HIT Systems

Streaming analytics is the process of ingesting and digesting live data from multiple data sources. In the healthcare domain, as the importance of extracting immediate insights while data are streaming into the system grows, the focus is shifting from batch processing to streaming analytics. With data increasing dramatically at high speeds, many informatics designs have been proposed to adapt healthcare domain into this new environment. In our previous work, we introduced a prototype of health informatics technology (HIT) framework that aims to address challenges in adopting state-of-the-art technologies to enable advanced healthcare analytic tasks in new streaming environments. We recently made major updates to the framework so that anomaly from multiple streaming data sources at different granularity levels can be detected in near real-time. In this paper, we detail the implementation and deployment of the framework in Kubernetes clusters and report its performances when tested on electronic health record (EHR) data of Veterans Affairs.

Park, Byung↗

ARM Climate Research Facility Quarterly Ingest Status Report

The purpose of this report is to provide a concise status update for ingests maintained by the Atmospheric Radiation Measurement (ARM) Climate Research Facility. The report is divided into the following sections: (1) new ingests for which development has begun, (2) progress on existing ingests, (3) future ingests that have been recently approved, (4) other work that leads to an ingest, and (5) top requested ingests from the ARM Data Archive. New information is highlighted in blue text.

54 ENVIRONMENTAL SCIENCES↗

ARM Climate Research Facility Quarterly Ingest Status Report

The purpose of this report is to provide a concise status update for ingests maintained by the Atmospheric Radiation Measurement (ARM) Climate Research Facility. The report is divided into the following sections: (1) new ingests for which development has begun, (2) progress on existing ingests, (3) future ingests that have been recently approved, (4) other work that leads to an ingest, and (5) top requested ingests from the ARM Data Archive. New information is highlighted in blue text.

54 ENVIRONMENTAL SCIENCES↗

File format for normalizing radiological concentration exposure rate and dose rate data for the effects of radioactive decay and weathering processes

This report specifies the electronic file format that was agreed upon to be used as the file format for normalized radiological data produced by the software tool developed under this TI project. The NA-84 Technology Integration (TI) Program project (SNL17-CM-635, Normalizing Radiological Data for Analysis and Integration into Models) investigators held a teleconference on December 7, 2017 to discuss the tasks to be completed under the TI program project. During this teleconference, the TI project investigators determined that the comma-separated values (CSV) file format is the most suitable file format for the normalized radiological data that will be outputted from the normalizing tool developed under this TI project. The CSV file format was selected because it provides the requisite flexibility to manage different types of radiological data (i.e., activity concentration, exposure rate, dose rate) from other sources [e.g., Radiological Assessment and Monitoring System (RAMS), Aerial Measuring System (AMS), Monitoring and Sampling). The CSV file format also is suitable for the file format of the normalized radiological data because this normalized data can then be ingested by other software [e.g., RAMS, Visual Sampling Plan (VSP)] used by the NA-84’s Consequence Management Program.

97 MATHEMATICS AND COMPUTING↗

Operating characteristics of the Langley Mach 7 Scramjet Test Facility

Operating characteristics of the Langley Mach 7 Scramjet Test Facility are described. The facility is designed for testing airframe integrated scramjet (supersonic combustion ramjet) engine models. Features include duplication of the flight Mach number total enthalpy, flight altitude simulation, and simulation of engine airframe integration effects such a bow shock wave precompression and boundary layer ingestion by the engine. Data obtained from facility calibration and from tests of a hydrogen burning, airframe integrated scramjet are discussed. An adverse interaction between the facility flow and the scramjet engine flow during combustion of the fuel is described.

Guy, R. W.↗

Effect of Background Pressure on the Plasma Oscillation Characteristics of the HiVHAc Hall Thruster

During a component compatibility test of the NASA HiVHAc Hall thruster, a number of plasma diagnostics were implemented to study the effect of varying facility background pressure on thruster operation. These diagnostics characterized the thruster performance, the plume, and the plasma oscillations in the thruster. Thruster performance and plume characteristics as functions of background pressure were previously published. This paper focuses on changes in the plasma oscillation characteristics with changing background pressure. The diagnostics used to study plasma oscillations include a high-speed camera and a set of high-speed Langmuir probes. The results show a rise in the oscillation frequency of the "breathing" mode with rising background pressure, which is hypothesized to be due to a shortening acceleration/ionization zone. An attempt is made to apply a simplified ingestion model to the data. The combined results are used to estimate the maximum acceptable background pressure for performance and wear testing.

Hall Thrusters↗

Effect of Background Pressure on the Plasma Oscillation Characteristics of the HiVHAc Hall Thruster

During a component compatibility test of the NASA HiVHAc Hall thruster, a high-speed camera and a set of high-speed Langmuir probes were implemented to study the effect of varying facility background pressure on thruster operation. The results show a rise in the oscillation frequency of the breathing mode with rising background pressure, which is hypothesized to be due to a shortening accelerationionization zone. An attempt is made to apply a simplified ingestion model to the data. The combined results are used to estimate the maximum acceptable background pressure for performance and wear testing.

Hall Thrusters↗