Search NASA⌕ Search

SEARCH · Search NASA

Results for “GES DISC”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14
All sourcesNASA NTRSDOE OSTI
NASA NTRS2017

Complexities in Subsetting Level 2 Data

Satellite Level 2 data presents unique challenges for tools and services. From nonlinear spatial geometry to inhomogeneous file data structure to inconsistent temporal variables to complex data variable dimensionality to multiple file formats, there are many difficulties in creating general tools for Level 2 data support. At NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), we are implementing a general Level 2 Subsetting service for Level 2 data to a user-specified spatio-temporal region of interest (ROI). In this presentation, we will unravel some of the challenges faced in creating this service and the strategies we used to surmount them.

Data acces↗
NASA NTRS2017

Use of Schema on Read in Earth Science Data Archives

Traditionally, NASA Earth Science data archives have file-based storage using proprietary data file formats, such as HDF and HDF-EOS, which are optimized to support fast and efficient storage of spaceborne and model data as they are generated. The use of file-based storage essentially imposes an indexing strategy based on data dimensions. In most cases, NASA Earth Science data uses time as the primary index, leading to poor performance in accessing data in spatial dimensions. For example, producing a time series for a single spatial grid cell involves accessing a large number of data files. With exponential growth in data volume due to the ever-increasing spatial and temporal resolution of the data, using file-based archives poses significant performance and cost barriers to data discovery and access. Storing and disseminating data in proprietary data formats imposes an additional access barrier for users outside the mainstream research community. At the NASA Goddard Earth Sciences Data Information Services Center (GES DISC), we have evaluated applying the schema-on-read principle to data access and distribution. We used Apache Parquet to store geospatial data, and have exposed data through Amazon Web Services (AWS) Athena, AWS Simple Storage Service (S3), and Apache Spark. Using the schema-on-read approach allows customization of indexing spatially or temporally to suit the data access pattern. The storage of data in open formats such as Apache Parquet has widespread support in popular programming languages. A wide range of solutions for handling big data lowers the access barrier for all users. This presentation will discuss formats used for data storage, frameworks with This presentation will discuss formats used for data storage, frameworks with support for schema-on-read used for data access, and common use cases covering data usage patterns seen in a geospatial data archive.

cloud applications↗
NASA NTRS2018

Application of Aura OMI L2G Products Compared with NASA MERRA-2 Assimilation

The Ozone Monitoring Instrument (OMI) is one of the instruments aboard NASA's Aura satellite. It measures ozone total column and vertical profile, aerosols, clouds, and trace gases including NO2, SO2, HCHO, BrO, and OClO using absorption in the ultraviolet electromagnetic spectrum (280 - 400 nm). OMI Level-2G (L2G) products are based on the pixel-level OMI granule satellite measurements stored within global 0.25 deg. X 0.25 deg. grids, therefore they conserve all the Level 2 (L2) spatial and temporal details for 24 hours of scientific data in one file. The second Modern-Era Retrospective analysis for Research and Applications (MERRA-2) is NASA's atmospheric reanalysis, using an upgraded version of Goddard Earth Observing System Model, version 5 (GEOS-5) data assimilation system. MERRA-2 includes aerosol data reanalysis and improved representations of stratospheric ozone, compared with its predecessor MERRA, in both instantaneous and time-averaged collections. It is found that simply comparing satellite Level-3 products might cause biases, due to lack of detailed temporal and original retrieval information. It is therefore preferable to inter-compare or implement satellite derived physical quantities directly with/to model assimilation with as high temporal and spatial resolutions as possible. This study will demonstrate utilization of OMI L2G daily aerosol and ozone products by comparing them with MERRA-2 hourly aerosol/ozone simulations, matched in both space and time aspects. Both OMI and MERRA-2 products are accessible online through NASA Goddard Earth Sciences Data Information Services Center (GES DISC, https://disc.gsfc.nasa.gov/).

OMI L2G products↗
NASA NTRS2014

Enhancement of Mutual Discovery, Search, and Access of Data for Users of NASA and GEOSS-Cataloged Data Systems

An ongoing NASA-funded Data Rods (time series) project has demonstrated the removal of a longstanding barrier to accessing NASA data (i.e., accessing archived time-step array data as point-time series) for selected variables of the North American and Global Land Data Assimilation Systems (NLDAS and GLDAS, respectively) and other NASA data sets. Data rods are pre-generated or generated on-the-fly (OTF), leveraging the NASA Simple Subset Wizard (SSW), a gateway to NASA data centers. Data rods Web services are accessible through the CUAHSI Hydrologic Information System (HIS) and the Goddard Earth Sciences Data and Information Services Center (GES DISC) but are not easily discoverable by users of other non-NASA data systems. An ongoing GEOSS Water Services project aims to develop a distributed, global registry of water data, map, and modeling services cataloged using the standards and procedures of the Open Geospatial Consortium and the World Meteorological Organization. Preliminary work has shown GEOSS can be leveraged to help provide access to data rods. A new NASA-funded project is extending this early work.

data access↗
NASA NTRS2018

Total and Spectral Solar Irradiance Sensor (TSIS) Project Status

TSIS-1 studies the Sun's energy input to Earth and how solar variability affects climate. TSIS-1 will measure both the total amount of light that falls on Earth, known as the total solar irradiance (TSI), and how that light is distributed among ultraviolet, visible and infrared wavelengths, called solar spectral irradiance (SSI). TSIS-1 will provide the most accurate measurements of sunlight and continue the long-term climate data record. TSIS-1 includes two instruments: the Total Irradiance Monitor (TIM) and the Spectral Irradiance Monitor (SIM), integrated into a single payload on the International Space Station (ISS). The TSIS-1 TIM and SIM instruments are upgraded versions of the two instruments that are flying on the Solar Radiation and Climate Experiment (SORCE) mission launched in January 2003. NASA Goddard's TSIS project responsibilities include project management, system engineering, safety and mission assurance, and engineering oversight for TSIS-1. TSIS-1 was installed on the International Space Station in December 2017. At the end of the 90-day commissioning phase, responsibility for TSIS-1 operations transitions to the Earth Science Mission Operations (ESMO) project at Goddard for its 5-year operations. NASA contracts with the University of Colorado Laboratory for Atmospheric and Space Physics (LASP) for the design, development and testing of TSIS-1, support for ISS integration, science operations of the TSIS-1 instrument, data processing, data evaluation, calibration and delivery to the Goddard Earth Science Data and Information Services Center (GES DISC).

TSIS↗
NASA NTRS2018

Restructuring Big Data to Improve Data Access and Performance in Analytic Services Making Research More Efficient for the Study of Extreme Weather Events and Application User Communities

By developing and enhancing various services and tools, the GES DISC provides users with the capability to access and visualize data, and to make comparisons of data from multiple sensor and models via a number of cross-discipline projects. Discovering Data via Faceted Web Interface Web interface to data products and services Search and Download mechanisms Dataset Landing Pages Accessing Data through Interoperable Services: GDS – GrADS Data Server OPeNDAP - Open-source Project for a Network Data Access Protocol WMS – OGC service GIS connector – allowing IS tools to access data easier (coming soon) HTTPS -- direct online access Downloading Data Basics: Subset and egridding Service – Parameter, Spatial, Time, Vertical, Mean averaging, format conversion, and regridding for L3/L4 gridded data Swath Data Subsetter – Parameter, spatial subset of L2 /L1 data. Visualizing Data Online: Giovanni –Visualization and Analysis L3/L4 gridded data AIRS NRT Viewer – AIRS near-real-time DQVis – L2 data quality visualization

data cube↗
NASA NTRS2018

Challenges in Development of Online Visualization and Analysis Tools for Satellite Data

Over the years, various online visualization and analysis tools have been developed to facilitate satellite data access and help scientific users around the world to conduct research and develop applications (e.g., data product evaluation, what-if questions, etc.). For those who are new to satellite data products, using them can be a daunting task due to many obstacles in data processing such as data formats, complex data structures, special software packages, unfamiliar terminology, etc., especially when one is not sure whether a dataset is suitable for his/er research project. Even for experienced users, developing software for data processing and analysis can be a costly and time-consuming task. Online visualization tools can overcome many of these difficulties and allow users to focus on scientific questions. For example, Giovanni (the Geospatial Interactive Online Visualization and Analysis Infrastructure, https://giovanni.gsfc.nasa.gov), developed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), allows access over 1900 satellite and model variables in 82 measurement groups of 8 disciplines without downloading data and software. Main features include basic functions for data analysis and visualization, data provenance, output data in different formats (ASCII, NetCDF, GeoTIFF), and more. Over the years, ~1700 peer-reviewed publications in different disciplines have been benefited from Giovanni in research activities (e.g. initial investigation, what-if questions, product evaluation). Despite the success of online visualization and analysis tools, challenges and new opportunities still exist and more can be done with new requirements and technology. Examples are: a) how to increase the efficiency of dataset search by enhancing intuitive aspects; b) how to facilitate interdisciplinary research; c) how to provide data quality information; d) how to engage users to participate in data quality assessment; and more. NASA Earth Observing System Data and Information System (EOSDIS) satellite-based data products are processed at various levels ranging from Level 0 to Level 4. While most users use data products at higher levels (Level-3 and 4), products at lower levels are still important for case studies, algorithm development, ground validation, etc. In this presentation, we will use Giovanni as an example to present and discuss challenges and near-future opportunities for satellite data online visualization and analysis tools.

Liu, Zhong↗
NASA NTRS2018

Enriching the Twitter Stream Increasing Data Mining Yield and Quality Using Machine Learning

Social media data streams are important sources of real-time and historical global information for science applications. At the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), we are exploring the Twitter data stream for its potential in augmenting the validation program of NASA Earth science missions, specifically the Global Precipitation Measurement (GPM) mission. We have implemented a tweet processing infrastructure that outputs classified precipitation tweets. Inputs are "passive" tweets, along with a smaller number of tweets from "active" participants, i.e., those knowingly contributing to our effort. The "active" tweets, presumably of higher quality, enrich the Twitter stream. "Active" sources include data scraped from other social media (e.g., public Facebook posts) and data from existing crowdsourcing programs (e.g., mPING reports). In addition, there is likely relevant precipitation information in images and documents that are the end points of links often included in tweets. Information derived from these "active" sources could then be tweeted into the Twitter stream, thus enriching its quality. The objective of our current work is to mine these tweet­ linked images and documents, using neural networks, to increase the information content and quality related to precipitation. For images, we classified them as either precipitation-related or not. For training and validation, we used images obtained via the Google custom search API. We created two models: (1) by training a simple Convolutional Neural Network and (2) by using transfer learning principles to adapt a pre-trained object recognition model. For documents, both those linked to tweets and the tweet contents, we trained Hierarchical Attention Networks to determine precipitation occurrence, type, and intensity. For training and validation, we used a keyword-filtered tweet data set labelled with ground truth data from Dark Sky (an API to retrieve weather-related labels) and the National Severe Storms Laboratory's Multi­ Radar/Multi-Sensor (MRMS) system. Our results demonstrated the efficacy of our machine learning approaches for enriching the Twitter stream, to derive information potentially useful for validation of earth science satellite data.

Albayrak, Arif↗
NASA NTRS2018

The Advantages of Synergy-Quantitative Earth Science Data Visualization and Analysis with Giovanni, Panoply, and Excel

The NASA Giovanni data analysis system provides a multitude of basic analysis capabilities for numerous Earth science data products which are available in the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) archive, as well as for additional selected data products provided by other NASA Distributed Active Archive Center (DAAC) archives. In Giovanni, users can easily generate time-averaged data maps, area-averaged time-series, Latitude-Time and Longitude-Time Hovmoeller diagrams, correlation maps, accumulation maps, and map animations (22 analysis options are available in total). While ASCII text output is available for time-series plots, it is not included as an option for data maps. In order to provide a quantitative, easy-to-use numerical output in ASCII text form, the NetCDF file output from a Giovanni visualization is downloaded and then opened with the free NASA visualization software package Panoply. Panoply provides the capability of translating the Giovanni file into comma-separated-variable (CSV) output. Panoply also provides additional visualization options, including the facile calculation of difference maps and quasi-anomaly maps using Giovanni output files. The CSV files from Panoply can then be imported into an Excel spreadsheet, where an Excel macro converts the CSV files. The output consists of latitude-longitude-data value triads in text form for maps, and either longitude-time-data value or latitude-time-data value triads in text form for Hovmoeller diagrams. This presentation will explicate the basic procedure for the conversion, and then provide several examples where the procedure is applied to Giovanni output from different analysis options.

Earth science↗
NASA NTRS2019

Online Visualization and Analysis of NASA Satellite-Based Global and Regional Precipitation Products Through Giovanni

Observational data are essential for Earth science research and applications. Traditional ground-based observations suffer from many limitations (e.g. costly deployment). As a result, data are often sparse and inconsistent, especially over vast oceans that cover nearly 71% of the Earth's surface, and for remote continents. Precipitation is one of the important physical parameters in the global hydrological cycle and other disciplines. Each year, severe floods and droughts happen in different parts of the world and cause significant damage to the economy, as well as human casualties (e.g. Hurricane Katrina, the Dust Bowl). Accurate and timely precipitation observations and predictions are important for research and applications. However, ground-based precipitation observations are quite limited, especially in remote and mountainous regions. Since the satellite era began, satellite-based precipitation products have gained popularity in Earth science research, applications, and education. Accessing satellite products can be a daunting task to many users, especially those who do not have prior experience or knowledge with satellite data. Recognizing this obstacle, the NASA Goddard Earth Sciences and Data and Information Services Center (GES DISC), home to data archives for the NASA-JAXA Tropical Rainfall Measuring Mission (TRMM) and Global Precipitation Measurement (GPM), has developed data services including an online visualization and analysis tool, Giovanni (the Geospatial Interactive Online Visualization ANd aNalysis Infrastructure), enabling users at different levels to access, explore, and evaluate NASA satellite-based data products without downloading either data and software, or requiring coding. Currently, global and regional precipitation products from different satellite missions (TRMM, GPM) and projects (e.g. the Modern Era Retrospective-analysis for Research and Applications Version 2 (MERRA-2), and the North American Land Data Assimilation System (NLDAS)), ranging from half-hourly to monthly temporal resolution, are available in Giovanni. There are over 1900 variables in Giovanni, covering measurements in precipitation, hydrology, atmospheric dynamics, atmospheric chemistry, etc. In this poster presentation, we will provide a live demonstration of Giovanni and its latest development, including precipitation-related variables, and new basic features such as polar projections. The session will also provide a Q&A opportunity for attendees.

Giovanni↗
NASA NTRS2019

Implementing Polar Projections with OGC Services for the Enhancement of AIRS NRT Visualization in LANCE

The Atmospheric Infrared Sounder (AIRS) NRT product is one important element in the Land, Atmosphere Near real-time Capability for EOS (LANCE). The LANCE processing of AIRS NRT products and the image generation are performed at the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC). The Open Geospatial Consortium (OGC) services are being utilized to access AIRS NRT images. The ongoing AIRS NRT imagery enhancement work includes adding a new set of the images in polar projections. Polar projections are commonly used for mapping Antarctica and Arctic regions. We have implemented more precise south polar (EPSG:3031) projection and north polar (EPSG:3413) projection making our OGC service instances more useful and interoperable. Thus, AIRS NRT data can be easily accessed and integrated with other applications. It greatly increases the impact of our data on researches in polar regions.In this presentation, we will introduce the optimized processing workflow for OGC services from data access with spatial-temporal index to data visualization with different SLD, and demonstrate how to use open source software to provide more precise map images in polar projections.

Zhao, Peisheng↗
NASA NTRS2019

Simplifying Satellite and Ground Data Validation with Level-2 Subsetting

We demonstrate that scientists can simplify their satellite data validation workflow with the use of NASA Godddard Earth Sciences Data and Information Services Center (GES DISC) subsetting services. We perform a sample validation of Aura ozone products collocated with ground-based ozone measurements using subsetting services to trim satellite data to only the relevant user-defined variables and spatio-temporal region. Because the subsetting service automatically returns only relevant data granules that adhere to a set of user-defined coincidence criteria, user workload is greatly reduced. Moreover, the resultant data files are substantially smaller than full data granules due to the subsetting service further culling the data to the relevant geospatio-temporal coincidence criteria, user-defined variables, and user-defined dimensions of variables. This decreases data download throughput and file storage requirements. The validation presented here quantifies the time and file size savings that can be achieved by utilizing subsetting services within the satellite data validation workflow.

Johnson, James↗
NASA NTRS2019

Streamlining Access to Satellite Level 2 Data

Users wishing to work with Satellite Level 2 data often have to overcome many hurdles: various file format standards, large files containing hundreds of variables, inconsistent presentation of geospatial and temporal variables, inhomogeneous data structures, and nonlinear spatial geometry. The Level 2 Subsetting service at NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) has many options to streamline and simplify user access to this data. In this presentation, we will detail the capabilities of this service and demonstrate how users can incorporate it into their Level 2 data workflow.

Huwe, Paul↗
NASA NTRS2019

Citizen Science Twitter Data Management for Earth Science Applications

Social media data can provide useful real-time and historical information relating to the natural world, but managing this data poses challenges. Scientists at GES DISC are exploring the potential of Twitter data to augment precipitation data from the Global Precipitation Measurement (GPM) mission. However, the format of Twitter data is unconventional in the context of NASA data centers, resulting in frustration for scientists who need to work with the data. This study investigated procedures and standards needed to properly manage Twitter data to make them compatible with these data centers. After comparing databases, the study found that the MongoDB database was best suited for the storage of raw Twitter data due to its flexibility, ability to be accessed by multiple users, and querying functionality. The study used the Python package Zarr to transform processed Twitter data into a gridded format similar to that of satellite data. Each Tweet was mapped onto a time-space grid; each grid location contained information about Tweet attributes and precipitation. The study developed a pipeline for downloading, storing, and gridding Twitter data and transformed Twitter data into an understandable format for users of NASA satellite data.

Li, Rachel↗
NASA NTRS2019

Cloud Giovanni: Reining in Costs and Improving Performance with Analytical Data Stores Using Scalable Serverless Architecture

Giovanni is the Geospatial Interactive Online Visualization ANd aNalysis Infrastructure developed at NASA GES DISC which provides a simple and intuitive way to visualize, analyze, and access vast amounts of Earth science data. It receives large number of user requests each day for a variety of analysis and visualization services, which leads to the big data challenge of serving gradually increasing large data volumes with diverse statistical algorithms. We hereby propose a multi-dimensional accumulation method which provides fast and cost-efficient cloud analysis for diverse services including both area averaging and time averaging. This method involves the weighted volume integration over multiple variable dimensions (time and space), and is implemented in AWS using Athena providing serverless and highly scalable data analysis. Compared to the standard method, this approach dramatically reduced the computational time by order of magnitude with a minimal AWS cost incurred. For example, for a benchmark of 10-year area averaging over the 1x1 degree daily variable, the computational time was reduced from minutes to seconds, and the Athena cost is only $5 for 100,000 requests.

Zhang, Hailiang↗
NASA NTRS2019

Bringing Analysis Closer to Data: Developing a Visualization Tool for L2 Earth Science Satellite Data

Earth Science satellite missions provide a unique opportunity for scientists to visualize complex and multifaceted observations projected geospatially across maps of the Earth. While visualization tools can help scientists comprehend, analyze, and share data, visualizing Level-2 Earth Sciences data poses its own specific set of challenges. Since the geospatial information in Level-2 data files is stored as independent variables, the plotting process involves matching dimensional information from latitude and longitude with a desired variable. Variables are stored in different ways across various Earth Science data file formats, which complicates the process of extracting data and plotting variables from a given file without requiring extensive user input and prerequisite familiarity with the file type variable structure. In coordination with NASA’s Goddard Earth Sciences Data Information Services Center (GES DISC), the team developed a Level-2 Earth Science data visualization tool that aims to address some of the complexities associated with plotting Level-2 data. This tool offers command-line and user interface support for file and variable selection to accommodate varying use cases and degrees of user familiarity with the structure of a given file. The visualization tool is written in Python 3 and utilizes a modular approach to facilitate continued expansion and reuse. In addressing some common complications involved in plotting Level-2 Earth Sciences data, the tool aims to help to link the process of analysis more directly with data acquisition and visualization, bringing analysis closer to data across levels of processing.

Li, Angela W.↗
NASA NTRS2019

Developing a Machine-Learning-Based Processing Framework for Twitter and Other Crowdsourced Data

Crowdsourced data streams such as Twitter and other social media are important sources of real-time and historical global information for Earth science applications. At the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), we have been exploring the Twitter data stream for its potential in augmenting the validation program of NASA's Global Precipitation Measurement (GPM) mission. To realize this potential, we need to increase the information density and enhance the quality of filtered precipitation tweets. We have implemented various components of a machine learning (ML)-based processing infrastructure for crowdsourced data that outputs, in this instance, useful and usable information derived from precipitation tweets. We have test enriched the Twitter stream with higher quality active tweets from those knowingly contributing to our effort and from existing crowdsourced programs (e.g., mPING, CoCoRaHS). We have experimented with various algorithms for processing tweets, including Naà ve Bayes, Convolutional Neural Network (CNN), Hierarchical Attention Network (HAN), and semi-supervised learning (with tri-training). Our current work focuses on (1) automated review of Earth science-related publications to determine relationships between discipline research needs and ML algorithms; (2) investigating Sequential Generative Adversarial Network (SeqGAN) for processing precipitation tweets for anomaly detection; and (3) managing crowdsourced data in a way that is compatible with existing NASA satellite data archives and using the data for ML applications. Key results include (1) network visualization of NLP-processed publications in various Earth science disciplines; (2) difference between GPM-linked, generated tweets and collected actual tweets that is small for GPM-determined light to moderate rain cases and high for GPM-determined heavy rain cases; and (3) identification of MongoDB for storing raw tweets and Zarr format for gridded tweets (compatible with GPM data). Our results have taken us a step closer to an operational ML-based tweet processing infrastructure and have already demonstrated that tweet-derived precipitation information is potentially useful for validation of Earth science satellite data.

Teng, William↗
NASA NTRSDate not supplied

Treating gridded geospatial data as point data to simplify analytics

Gridded geospatial remote sensing (satellite) data has traditionally been stored in file-based multidimensional arrays to preserve the locality of data. Measurements from locations that are physically next to each other on earth remain next to each other in the arrays. Maintaining this locality is useful when running calculations like reprojection, but unnecessary for many other calculations. This talk will go through a real world example of a tool redesign at the Goddard Earth Sciences Data and Information Services Center (GES DISC), showing the advantages of using the data frame model for calculating summary statistics, where measurement proximity is unimportant.

Analysis-ready data↗