Search NASA⌕ Search

SEARCH · Search NASA

Results for “data ingest”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19

GLDAS-2 Land Surface Model Data and Data Services at NASA GES DISC

The goal of the NASA Global Land Data Assimilation System (GLDAS, https://ldas.gsfc.nasa.gov/gldas(https://ldas.gsfc.nasa.gov/gldas)) is to generate optimal fields of land surface states and fluxes by ingesting satellite- and ground-based observational data products, using advanced land surface modeling and data assimilation techniques (Rodell et al., 2004).The GLDAS dataset currently archived at and distributed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC, https://disc.gsfc.nasa.gov/ (https://disc.gsfc.nasa.gov/)) is GLDAS Version 2 (GLDAS-2). It contains a series of output fields from the upgraded Noah-3.6, Catchment-F2.5, and VIC-4.1.2 Land Surface Models (LSMs) in the Land Information System (LIS-V7, https://lis.gsfc.nasa.gov/ (https://lis.gsfc.nasa.gov/)). GLDAS-2 has three components:GLDAS-2.0, GLDAS-2.1, and GLDAS-2.2. GLDAS-2.0 is forced entirely with the upgraded Princeton Meteorological ForcingV2.2 Dataset and provides a temporally consistent series from 1948 through 2014. GLDAS-2.1 is forced with a combination of model and observation data, with data spanning from 2000 to the present. The GLDAS-2.2 product suite uses data assimilation(DA), whereas the GLDAS-2.0 and GLDAS-2.1 products are "open-loop" (i.e., no data assimilation). The choice of forcing data, as well as DA observation source, variable, and scheme, varies for different GLDAS-2.2 products. The currently availableGLDAS-2.2 data contain a daily 0.25-degree output from the Catchment-F2.5 LSM in LIS-V7. The data are forced with the meteorological analysis fields from the operational European Centre for Medium-Range Weather Forecasts Integrated Forecasting System (ECMWF-IFS) and assimilated with GRACE and GRACE-FO data, ranging from February 1, 2003 to the present. The current GLDAS-2.0 and 2.1 Noah LSM data were reprocessed in November 2019 and January 2020 respectively and their data from Catchment and VIC LSMs are new to the GLDAS-2 collection. This presentation provides a summary of theGLDAS-2 data products, their land surface fields, and their related data services at the GES DISC; and a description of the majorGLDAS-2 climatological characteristics as well as the intercomparison with the data of the previous version.

Hydrology↗

aosacsm.b1

ACSM instrument measure mass loading (ug/m3) of non-refractory aerosol particles particularly organics, sulfate, chlorides, ammonium, and nitrates in real-time. The instrument is run autonomously at various ARM sites at time resolution equal to 30 minutes. Different data products are developed. The raw data (.itx and .asc) files are ingested to produce the.a0 product - the first netCDF file. These raw files are used by the mentor to calculate the mentor processed mass loadings. The .a0 files are used to develop the .b1 level product. The .b1 level data is further integrated with mentor processed data to produce .c0 level data product.

54 ENVIRONMENTAL SCIENCES↗

Advances Towards Ingestion of Multi-Instrument Cloud-Cleared Infrared Radiances in a Global Data Assimilation and Forecast Framework

The progress towards the assimilation of hyperspectral infrared radiances affected by clouds in the NASA GEOS system is presented. After positive published results describing the assimilation of adaptively thinned AIRS cloud-cleared radiances in the 3DVAR GEOS, this team has then successfully tested adaptively thinned CrIS and IASI data as well, and then moved to assimilation in a hybrid 4DEnVAR configuration. The latest effort is focused on the creation of a portable, customizable version of the cloud-clearing algorithm that is being tailored for the hybrid 4DEnVAR system.

Reale, Oreste↗

Integrating Wind Profiling Radars and Radiosonde Observations with Model Point Data to Develop a Decision Support Tool to Assess Upper-Level Winds for Space Launch

On the day-of-launch, the 45th Weather Squadron (45 WS) Launch Weather Officers (LWOs) monitor the upper-level winds for their launch customers to include NASA's Launch Services Program and NASA's Ground Systems Development and Operations Program. They currently do not have the capability to display and overlay profiles of upper-level observations and numerical weather prediction model forecasts. The LWOs requested the Applied Meteorology Unit (AMU) develop a tool in the form of a graphical user interface (GUI) that will allow them to plot upper-level wind speed and direction observations from the Kennedy Space Center (KSC) 50 MHz tropospheric wind profiling radar, KSC Shuttle Landing Facility 915 MHz boundary layer wind profiling radar and Cape Canaveral Air Force Station (CCAFS) Automated Meteorological Processing System (AMPS) radiosondes, and then overlay forecast wind profiles from the model point data including the North American Mesoscale (NAM) model, Rapid Refresh (RAP) model and Global Forecast System (GFS) model to assess the performance of these models. The AMU developed an Excel-based tool that provides an objective method for the LWOs to compare the model-forecast upper-level winds to the KSC wind profiling radars and CCAFS AMPS observations to assess the model potential to accurately forecast changes in the upperlevel profile through the launch count. The AMU wrote Excel Visual Basic for Applications (VBA) scripts to automatically retrieve model point data for CCAFS (XMR) from the Iowa State University Archive Data Server (http://mtarchive.qeol.iastate.edu) and the 50 MHz, 915 MHz and AMPS observations from the NASA/KSC Spaceport Weather Data Archive web site (http://trmm.ksc.nasa.gov). The AMU then developed code in Excel VBA to automatically ingest and format the observations and model point data in Excel to ready the data for generating Excel charts for the LWO's. The resulting charts allow the LWOs to independently initialize the three models 0-hour forecasts against the observations to determine which is the best performing model and then overlay the model forecasts on time-matched observations during the launch countdown to further assess the model performance and forecasts. This paper will demonstrate integration of observed and predicted atmospheric conditions into a decision support tool and demonstrate how the GUI is implemented in operations.

Bauman, William H., III↗

A Scalability Model for ECS's Data Server

This report presents in four chapters a model for the scalability analysis of the Data Server subsystem of the Earth Observing System Data and Information System (EOSDIS) Core System (ECS). The model analyzes if the planned architecture of the Data Server will support an increase in the workload with the possible upgrade and/or addition of processors, storage subsystems, and networks. The approaches in the report include a summary of the architecture of ECS's Data server as well as a high level description of the Ingest and Retrieval operations as they relate to ECS's Data Server. This description forms the basis for the development of the scalability model of the data server and the methodology used to solve it.

Menasce, Daniel A.↗

The Effects of Vortex Ingestion on the Flow in a Diffusing S-duct

An experimental study of the effects of an ingested vortex on the flowfield of a diffusing S-duct is reported. The vortex is generated through the use of a stationary pinwheel device mounted upstream of the diffusing S-duct. Three test conditions vary the location of the vortex in the duct inlet crossplane. For each condition of ingested vortex, a baseline S-duct and an S-duct with an array of vortex generators is examined. The data taken consist of duct inlet and exit crossplane surveys of velocity and total pressure. Duct surface flow visualization and static pressure are also recorded. The data acquired in these tests are compared to identical S-duct data taken in the absence of the ingested vortex. The ingested vortex is observed to have a strong influence on the flowfield inside (and exiting) the S-duct, but only when the vortex impinges at the inlet crossplane location coincident with the crossplane location of downstream flow separation within the duct. When the ingested vortex impinges at this location it reduces the extent of flowfield separation inside the baseline duct and promotes stronger crossflow in the exit plane of both the baseline duct and the duct with installed vortex generators. This enhanced crossflow also strengthens the vortices shed from the vortex generators. The other impingement locations of the ingested vortex are found to produce little effect on the flowfield of the duct with or without vortex generators.

Wendt, B. J.↗

Effect of fluid ingestion on orthostatic responses following acute exercise

Orthostatic tolerance is impaired following an acute bout of exercise. This study examined the effect of fluid ingestion following treadmill exercise in restoring the cardiovascular responses to an orthostatic stress. Five men (age, 29.6 +/- 3.4 yrs) were exposed to a graded lower body negative (LBNP) pressure protocol (0 to -50 mmHg) during euhydration without exercise (C), 20 minutes after exercise dehydration (D), 20 minutes after exercise and fluid ingestion (FI20), and 60 minutes after exercise and fluid ingestion (FI60). Fluid ingestion (mean +/- SE) consisted of water-ingestion equivalent to 50% of the body weight lost during exercise (520 +/- 15 ml). Exercise dehydration resulted in significantly higher heart rates (119 +/- 8 vs 82 +/- 7 bpm), lower systolic blood pressures (95 +/- 1.7 vs 108 +/- 2.3 mmHg), a smaller increase in leg circumference (3.7 +/- 4 vs 6.9 +/- 1.0 mm), and an attenuated increase in total peripheral resistance (2.58 +/- 1.2 vs 4.28 +/- 0.9 mmHg/L/min) at -50 mmHg LBNP compared to the C condition. Fluid ingestion (both 20 and 60), partially restored the heart rate, systolic blood pressure, and total peripheral resistance responses to LBNP, but did not influence the change in leg circumference during LBNP (4 +/- 0.3 for R20 and 2.8 +/- 0.4 mm for R60). These data illustrate the effectiveness of fluid ingestion on improving orthostatic responses following exercise, and suggest that dehydration is a contributing factor to orthostatic intolerance following exercise.

NASA Center JSC↗

Berkeley Nuclear Data Cloud (BDC) v2.3

The software was developed to serve and curate arbitrarily large datasets comprising data acquired from various mobile platforms. The software is contained in a number of server and client libraries. The former manage the ingestion, indexing, querying, and serving of the data. The latter libraries are distributed for Linux, Mac OSX, and Windows and enable users to interact with data downloaded from the service either in the form of an HDF5 file or streamed in a BSON data chunk. Using the Berkeley Data Cloud, researchers from varying fields can collaborate, compare results and curate both their raw data and the derived products of their analysis.

Cholia, Shreyas↗

OpenSAMPL: An Open Source Library for Timing and Synchronization Measurements and Analytics

Today's power grid operators are implementing timing and synchronization solutions that provide resilience to Global Navigation Satellite System (GNSS) vulnerabilities. These vendor-specific solutions often come with additional software applications that are designed to monitor that vendor's synchronization performance data. However, resilient timing architectures often resulting in multi-vendor solutions, including approaches that blend terrestrial clocks with space-based subscription services. In such an environment, collecting, analyzing, and visualizing data from a variety of sources within a single platform was heretofore not possible. To address this need, the US Department of Energy's Center for Alternative Synchronization and Timing (CAST) developed OpenSAMPL, the Open Synchronized Analytics and Monitoring Platform, an open-source Python framework for processing, loading, and observing clock measurement data from distributed devices. OpenSAMPL enables the ingestion of diverse clock-probe sources into a scalable time-series database and applies robust analytics. OpenSAMPL currently supports two vendor data pipelines, and will be extended to more in the near future, enabling seamless monitoring of a variety of timing and synchronization devices in a common environment.

Grant, Josh [ORNL] (ORCID:0000000163475060)↗

A satellite data processing and analysis software system for earth's atmosphere and surface research

The OASIS (Oceanic and Atmospheric Satellite Imaging System) is a satellite data processing and analysis software system being developed by the California Space Institute (Cal Space) for support of interdisciplinary and integrated earth sciences research programs. The system's software applications are integrated under a common executive, NASA's Transportable Application Executive (TAE). In this paper, TAE and the system software and hardware are described, and specific techniques used for ingesting, processing, analyzing, and graphically displaying data from many of the sensors presently being flown are presented. Scientific uses of these capabilities that are, or will shortly be, running under TAE at Cal Space are described.

Dealy, B.↗

A Performant, Scalable Processing Pipeline for High‐Quality and FAIR Environmental Sensor Data

High-resolution environmental monitoring is necessary to record, understand, and predict biogeochemical and ecological changes particularly in coastal systems but brings significant challenges in processing and making rapidly available the resulting data. The COMPASS-FME project established a network of coastal observational sites across the Chesapeake Bay and western Lake Erie regions extensively instrumented with soil, vegetation, and weather sensors logging data every 15 min. Our data processing framework, written in R and completely open source, prioritizes rapid model-experiment iteration and makes biogeochemical data rapidly available for quality assurance/quality control, analysis, and model ingestion. This pipeline is distinguished by a standardized and modular approach to data curation, extensive metadata and documentation, and its high performance. These attributes combine to make biogeochemical data rapidly accessible across COMPASS-FME and the broader community. Flexible, powerful, and reproducible approaches to handling high-volume environmental data are crucial for accelerating biogeosciences research.

Pennington, Stephanie C. [Pacific Northwest Nation↗

A New Tropical Cyclone Dynamic Initialization Technique Using High Temporal and Spatial Density Atmospheric Motion Vectors and Airborne Field Campaign Data

Background: Initialization of tropical cyclones in numerical weather prediction (NWP) systems is a great challenge: Mass-wind field balance; Secondary circulation and heating; Asymmetries. There can be large adjustments in structure and intensity in the first 24 hours if the initial vortex is not in balance: Spurious gravity waves; Spin-up (model and physics). Existing mesoscale NWP model TC (Tropical Cyclone) initialization strategies: Bogus vortex, cold start from global analyses; 3DVAR or 4DVAR, possibly with synthetic observations; EnKF (Ensemble Kalman Filter); Dynamic initialization. Dynamic initialization allows vortex to have improved balance and physics spin-up at the initial time (e.g., Hendricks et al. 2013, 2011; Nguyen and Chen 2011; Fiorino and Warner 1981; Hoke and Anthes 1976). Himawari-8 geostationary satellite has capability of continuous imagery (10-minutes) over the full disk: New GOES-R satellites will have same capability. This will allow for unprecedented observations of tropical cyclones. However, current data assimila1on systems are not capable of ingesting such high temporal observations (Atmospheric Mo1on Vectors - AMVs). Hourly AMVs are produced, and thinned to 100-kilometer spacing in the horizontal. An entirely new data assimilation concept is required to utilize these observations.

remote sensing↗

Machine-Learning-Based Adaptive Thinning of CrIS Radiances to Improve Global Tropical Cyclone Analysis and Forecasts

This work is focused on optimizing the assimilation of hyperspectral infrared (IR) radiances from the Cross-track Infrared Sounder (CrIS) with the goal of improving the representation of tropical cyclones (TCs) in global analyses and forecasts. Current operational assimilation systems rely on subsampling IR radiances on a regular thinning grid. A new and improved adaptive methodology based on machine learning (ML) recognizes TCs from geostationary satellite imagery and is implemented in the Goddard Earth Observing System (GEOS) model and data assimilation framework. The ML methodology is extensively trained on existing TC data sets and creates for each TC a dynamic mask, based on the evolving shape and life cycle of that specific event. Once a TC mask is created, a switch is then activated in the data assimilation system to alter the thinning, ingesting more CrIS radiances within the moving mask, thus increasing the TC sampling. After the TC dissipates, the assimilation of CrIS radiances reverts to normal data density. Results of TC segmentation provided by a state-of-the-art generative machine learning model known as the Denoising Diffusion Probabilistic Model (DDPM) are compared to the previously used U-Net model. The new approach surpasses the performance of the previously developed one. The methodology is applied to both clear-sky and cloud-cleared radiances. Benefits from the latter methodology, particularly in improving the structure of TCs and the intensity forecasts, are presented.

Oreste Reale↗

GES DISC Data Recipes in Jupyter Notebooks

The Earth Science Data and Information System (ESDIS) Project manages twelve Distributed Active Archive Centers (DAACs) which are geographically dispersed across the United States. The DAACs are responsible for ingesting, processing, archiving, and distributing Earth science data produced from various sources (satellites, aircraft, field measurements, etc.). In response to projections of an exponential increase in data production, there has been a recent effort to prototype various DAAC activities in the cloud computing environment. This, in turn, led to the creation of an initiative, called the Cloud Analysis Toolkit to Enable Earth Science (CATEES), to develop a Python software package in order to transition Earth science data processing to the cloud. This project, in particular, supports CATEES and has two primary goals. One, to transition data recipes created by the Goddard Earth Science Data and Information Service Center (GES DISC) into an interactive and educational environment using JupyterNotebooks. Two, to acclimate Earth scientists to cloud computing. To accomplish these goals, we create JupyterNotebooks to compartmentalize the different steps of data analysis and help users obtain and parse data from the command line. We also develop a Docker container, comprised of Jupyter Notebooks, Python dependencies, and command line tools, and configure it into an easy-to-deploy package. The end result is an end-to-end product that simulates the use case of end users working in the cloud computing environment.

discoverability↗

Global Data Assembly Center (GDAC) report to the GHRSST Science Team

In 2017-2018 the Global Data Assembly Center (GDAC) at NASA’s Physical Oceanography Distributed Active Archive Center (PO.DAAC) provided ingest, archive, distribution and user services for GHRSST operational data streams with improved and evolved tools, services, and tutorials and interfaced with the user community to address technical inquiries. The GDAC provided access to new GHRSST datasets as well retired a significant number of deprecated GHRSST datasets from its discovery services. The following sections summarize and document the specific achievements of the GDAC to the GHRSST community.

Finch, Chris↗

Global Data Assembly Center (GDAC) Report to the GHRSST Science Team

In 2017-2018 the Global Data Assembly Center (GDAC) at NASA’s Physical Oceanography Distributed Active Archive Center (PO.DAAC) provided ingest, archive, distribution and user services for GHRSST operational data streams with improved and evolved tools, services, and tutorials and interfaced with the user community to address technical inquiries. The GDAC provided access to new GHRSST datasets as well retired a significant number of deprecated GHRSST datasets from its discovery services. The following sections summarize and document the specific achievements of the GDAC to the GHRSST community.

Finch, Chris↗

Global Data Assembly Center (gdac) Report to the GHRSST Science Team

In 2019-2020 the Global Data Assembly Center (GDAC) at NASA’s Physical Oceanography Distributed Active Archive Center (PO.DAAC) provided ingest, archive, distribution and user services for GHRSST operational data streams with improved and evolved tools, services, and tutorials, and interfaced with the user community to address technical inquiries. Several new GHRSST datasets including reprocessed MODIS Aqua/Terra L2P with improved cloud screening were made available. The PO.DAAC participated in the evolving GHRSST data management re architecture and development activities. GHRSST Science Team member Edward Armstrong assumed the role of co-leader of the CEOS SST Virtual Constellation. The following sections summarize and document the specific achievements of the GDAC to the GHRSST community.

Finch, Chris↗

Deep Dive Topic: Analysis

This section focuses on analysis of I/O behavior and the feedback resulting from that analysis. We define analysis of I/O behavior as interpreting and deriving meaning from I/O instrumentation data. We define feedback as the product of the analysis process. Feedback can be thought of as the link between analysis and the subsequent response to that analysis, such as I/O tuning or policy decisions (see Section 4.4).Analysis and feedback may be consumed by either human stakeholders or machine algorithms. If it is to be consumed by human stakeholders, then the analysis process will likely emphasize understandable language and visualization techniques. If it is to be consumed by machine algorithms, then it will likely emphasize the use of machine-parsable, consistent formats that are suitable for ingestion by policy engines and data-mining tools.

Snyder, Shane↗