Search NASASearch

SEARCH · Search NASA

Results for “HDF5”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

87 records · Page 5

A-Train Data Depot (ATDD)

ATDD is a web based tool which provides collocated data and display products for a number of A-train instruments Cloudsat, Calipso, OMI, AIRS, MODIS, MLS, POLDER-3, and ECWMF model data. Products provided include Clouds, Aerosols, Water Vapor, Temperatures and trace gases. All input data is online and in HDF4, HDF5 format. Display products include curtain images, horizontal strips, line plot overlays, and GE kmz files. Sample products are shown for two type of events. Hurricane event, Norbert, Oct 8, 2008 and a dust storm event over the Arabian Sea, Nov 13-14, 2008.

Smith, Peter M.

NASA Tech Briefs, June 2013

Topics include: Cloud Absorption Radiometer Autonomous Navigation System - CANS, Software Method for Computed Tomography Cylinder Data Unwrapping, Re-slicing, and Analysis, Discrete Data Qualification System and Method Comprising Noise Series Fault Detection, Simple Laser Communications Terminal for Downlink from Earth Orbit at Rates Exceeding 10 Gb/s, Application Program Interface for the Orion Aerodynamics Database, Hyperspectral Imager-Tracker, Web Application Software for Ground Operations Planning Database (GOPDb) Management, Software Defined Radio with Parallelized Software Architecture, Compact Radar Transceiver with Included Calibration, Software Defined Radio with Parallelized Software Architecture, Phase Change Material Thermal Power Generator, The Thermal Hogan - A Means of Surviving the Lunar Night, Micromachined Active Magnetic Regenerator for Low-Temperature Magnetic Coolers, Nano-Ceramic Coated Plastics, Preparation of a Bimetal Using Mechanical Alloying for Environmental or Industrial Use, Phase Change Material for Temperature Control of Imager or Sounder on GOES Type Satellites in GEO, Dual-Compartment Inflatable Suitlock, Modular Connector Keying Concept, Genesis Ultrapure Water Megasonic Wafer Spin Cleaner, Piezoelectrically Initiated Pyrotechnic Igniter, Folding Elastic Thermal Surface - FETS, Multi-Pass Quadrupole Mass Analyzer, Lunar Sulfur Capture System, Environmental Qualification of a Single-Crystal Silicon Mirror for Spaceflight Use, Planar Superconducting Millimeter-Wave/Terahertz Channelizing Filter, Qualification of UHF Antenna for Extreme Martian Thermal Environments, Ensemble Eclipse: A Process for Prefab Development Environment for the Ensemble Project, ISS Live!, Space Operations Learning Center (SOLC) iPhone/iPad Application, Software to Compare NPP HDF5 Data Files, Planetary Data Systems (PDS) Imaging Node Atlas II, Automatic Calibration of an Airborne Imaging System to an Inertial Navigation Unit, Translating MAPGEN to ASPEN for MER, Support Routines for In Situ Image Processing, and Semi-Supervised Eigenbasis Novelty Detection.

Source record

Global Precipitation Measurement (GPM) Mission: Precipitation Processing System (PPS) GPM Mission Gridded Text Products Provide Surface Precipitation Retrievals

In February 2015, the Global Precipitation Measurement (GPM) mission core satellite will complete its first year in space. The core satellite carries a conically scanning microwave imager called the GPM Microwave Imager (GMI), which also has 166 GHz and 183 GHz frequency channels. The GPM core satellite also carries a dual frequency radar (DPR) which operates at Ku frequency, similar to the Tropical Rainfall Measuring Mission (TRMM) Precipitation Radar, and a new Ka frequency. The precipitation processing system (PPS) is producing swath-based instantaneous precipitation retrievals from GMI, both radars including a dual-frequency product, and a combined GMIDPR precipitation retrieval. These level 2 products are written in the HDF5 format and have many additional parameters beyond surface precipitation that are organized into appropriate groups. While these retrieval algorithms were developed prior to launch and are not optimal, these algorithms are producing very creditable retrievals. It is appropriate for a wide group of users to have access to the GPM retrievals. However, for researchers requiring only surface precipitation, these L2 swath products can appear to be very intimidating and they certainly do contain many more variables than the average researcher needs. Some researchers desire only surface retrievals stored in a simple easily accessible format. In response, PPS has begun to produce gridded text based products that contain just the most widely used variables for each instrument (surface rainfall rate, fraction liquid, fraction convective) in a single line for each grid box that contains one or more observations.This paper will describe the gridded data products that are being produced and provide an overview of their content. Currently two types of gridded products are being produced: (1) surface precipitation retrievals from the core satellite instruments GMI, DPR, and combined GMIDPR (2) surface precipitation retrievals for the partner constellation satellites. Both of these gridded products are generated for a.25 degree x.25 degree hourly grid, which are packaged into daily ASCII (American Standard Code for Information Interchange) files that can downloaded from the PPS FTP (File Transfer Protocol) site. To reduce the download size, the files are compressed using the gzip utility.This paper will focus on presenting high-level details about the gridded text product being generated from the instruments on the GPM core satellite. But summary information will also be presented about the partner radiometer gridded product. All retrievals for the partner radiometer are done using the GPROF2014 algorithmusing as input the PPS generated inter-calibrated 1C product for the radiometer.

precipitation

Improving the Accessibility and Use of NASA Earth Science Data

Many of the NASA Langley Atmospheric Science Data Center (ASDC) Distributed Active Archive Center (DAAC) multidimensional tropospheric and atmospheric chemistry data products are stored in HDF4, HDF5 or NetCDF format, which traditionally have been difficult to analyze and visualize with geospatial tools. With the rising demand from the diverse end‐user communities for geospatial tools to handle multidimensional products, several applications, such as ArcGIS, have refined their software. Many geospatial applications now have new functionalities that enable the end user to: Store, serve, and perform analysis on each individual variable, its time dimension, and vertical dimension. Use NetCDF, GRIB, and HDF raster data formats across applications directly. Publish output within REST image services or WMS for time and space enabled web application development. During this webinar, participants will learn how to leverage geospatial applications such as ArcGIS, OPeNDAP and ncWMS in the production of Earth science information, and in increasing data accessibility and usability.

Tisdale, Matthew

HDF Update

The HDF Group maintains and evolves HDF software used by NASA ESDIS program to manage remote sense data. In this talk we will discuss new features of HDF (Virtual Datasets, Single writerMultiple reader access, Community supported HDF5 compression filters) that address storage and IO performance requirements of the applications that work with the ESDIS data products.

Software evolution

Interactive Visualization of Near Real Time and Production Global Precipitation Measurement (GPM) Mission Data Online Using CesiumJS

Advancements in the capabilities of JavaScript frameworks and web browsing technology make online visualization of large geospatial datasets viable. Commonly this is done using static image overlays, prerendered animations, or cumbersome geoservers. These methods can limit interactivity andor place a large burden on server-side post-processing and storage of data. Geospatial data, and satellite data specifically, benefit from being visualized both on and above a three-dimensional surface. The open-source JavaScript framework CesiumJS, developed by Analytical Graphics, Inc., leverages the WebGL protocol to do just that. It has entered the void left by the abandonment of the Google Earth Web API, and it serves as a capable and well-maintained platform upon which data can be displayed. This paper will describe the technology behind the two primary products developed as part of the NASA Precipitation Processing System STORM website: GPM Near Real Time Viewer (GPMNRTView) and STORM Virtual Globe (STORM VG). GPMNRTView reads small post-processed CZML files derived from various Level 1 through 3 near real-time products. For swath-based products, several brightness temperature channels or precipitation-related variables are available for animating in virtual real-time as the satellite-observed them on and above the Earths surface. With grid-based products, only precipitation rates are available, but the grid points are visualized in such a way that they can be interactively examined to explore raw values. STORM VG reads values directly off the HDF5 files, converting the information into JSON on the fly. All data points both on and above the surface can be examined here as well. Both the raw values and, if relevant, elevations are displayed. Surface and above-ground precipitation rates from select Level 2 and 3 products are shown. Examples from both products will be shown, including visuals from high impact events observed by GPM constellation satellites.

satellite precipitation measurement

Seven Keys for Practical Understanding and Use of CGNS

We present key features of the CGNS standard, focusing on its two main elements, the data model (CGNS/SIDS) and its implementations (CGNS/HDF5 and CGNS/Python). The data model is detailed to emphasize how the topological user oriented information, such as families, are separated from the actual meshing that could be split or modified during the CFD work flow, and how this topological information is traced during the meshing process. We also explain why the same information can be described in multiple ways and how to handle such alternatives in an application. Two implementations, using HDF5 and Python, are illustrated in several use examples, both for archival and interoperability purposes. The CPEX extension formalized process is explained to show how to add new features to the standard in a consensual way; we present some of the next extensions to come. Finally we conclude by showing how powerful a consensual public approach like CGNS can be, as opposed to a stand-alone private one. All throughout the paper, we demonstrate how the use of CGNS could be of great benefit for both the meshing and CFD solver communities.

Poinot, Marc

Using Cloud-Based Storage Technologies for Earth Science Data

Cloud based infrastructure may offer several key benefits of scalability, built in redundancy and reduced total cost of ownership as compared with a traditional data center approach. However, most of the tools and software systems developed for NASA data repositories were not developed with a cloud based infrastructure in mind and do not fully take advantage of commonly available cloud-based technologies. Object storage services are provided through all the leading public (Amazon Web Service, Microsoft Azure, Google Cloud, etc.) and private (Open Stack) clouds, and may provide a more cost-effective means of storing large data collections online. We describe a system that utilizes object storage rather than traditional file system based storage to vend earth science data. The system described is not only cost effective, but shows superior performance for running many different analytics tasks in the cloud. To enable compatibility with existing tools and applications, we outline client libraries that are API compatible with existing libraries for HDF5 and NetCDF4. Performance of the system is demonstrated using clouds services running on Amazon Web Services.

Data

Microphone Phased Array NetCDF/HDF5 Archival Files: Application Program Interface Reference

An application program interface (API) has been developed for the creation and access of structured data files generated by microphone phased arrays utilized in aeroacoustics research. Two structured binary file formats are supported, namely NetCDF (Network Common Data Form) and HDF5 (Hierarchical Data Format) files. The API consists of a library of routines callable from C, Fortran or Matlab, with native versions of the API provided for each language. The libraries are divided into categories for file handling, file definition and initialization, data writing, data recovery, and error handling. The API is intended to provide a mechanism for generating self-describing binary files for long-term archiving of raw and processed data generated by phased array systems.

Humphreys, William M., Jr.

Understanding Machine Learning in Earth Science: A Natural Language Processing Approach

Machine learning (ML) is being increasingly utilized in Earth science research. Benefits of ML include efficiency, reduction of human error, and ability to extract hidden patterns within data. However, the mutual lack of each other’s domain knowledge by ML and Earth science stands as a barrier to timely and effective implementation. Earth science, in particular, faces challenges in generating sample data, compared to those of traditional ML problems such as face recognition or stock predictions, where data is abundant and not lacking in ground truth, which is necessary for labeling. Earth science data are more varying in formats, such as HDF5 and image resolutions, and are not standardized across instruments, even within a given Earth science discipline. Previous studies have been done to outline the specific challenges that Earth science faces with ML, while others have focused on using existing publications to mine information efficiently. Other resources such as Scikit-Learn have developed decision trees for choosing appropriate machine learning algorithms, but application within Earth science subjects becomes much more complex. For the current study, we propose a methodology and tool that aids in implementation of ML in Earth science using natural language processing (NLP). Our work comprises three main parts: (1) analyzing existing publications related to ML and Earth science, using natural language processing: (2) extracting from the publications information on ML models subjects in Earth Science: and (3) visualizing the extracted relationships as a network graph. The resulting network graph should aid the Earth science communities in applying optimal ML algorithms and guiding data preparation through visualization of similar studies. The network graph and analysis of document similarity will be the basis of our next step, which is to develop a decision tree for selecting optimal machine learning methodologies for specified Earth science applications.

Zheng, Laura

Assessing the Needs of NASA's Near Real-Time Earth Observation Products

"The 2017-2027 Decadal Survey for Earth Science and Applications from Space stated that NASA's Earth Science with planned implementation of applications provides sustained earth observations for societal benefits [1]. The Decadal Survey indicated that data latency is invaluable for time-sensitive applications including disaster risk reduction, wildland fire carbon emissions quantification, real-time measurements of the state of the hydrologic systems and many more. Data latency refers to the time between earth observation and data products available to users. During the past 13 years, NASA's Land, Atmosphere Near Real-Time Capability for Earth Observing Systems (LANCE) continues to provide free access to earth observation products that are made available much quicker than routine processing allows. The latency of most LANCE data products is Near Real-time (NRT) which is defined as less than three hours from satellite observations [2]. LANCE is managed by the Earth Science Data and Information System (ESDIS) Project at NASA Goddard Space Flight Center [3], and a User Working Group (UWG) is responsible for providing guidance to LANCE. LANCE data are used by direct users and brokers who add value to the data [4]. NASA Earth Applied Sciences Program (ASP) is one of the primary users of LANCE, which collaborates with partner organizations and provides support to scientists to solve problems in applications of earth observations. ASP promotes the use of LANCE NRT data products to demonstrate applications in decision making, facilitates end-user feedback to the science team to improve data products, and provides information on future demands for research. LANCE supports applications that need a rapid response including detecting wildland fires and volcanic eruptions, tracking smoke, ash and dust plumes, monitoring air quality and tracking extreme weather events such as hurricanes, landslides, and floods. To gather feedback regarding the availability, accessibility and actionability of NASA's NRT data products for societal benefit, three surveys and a few discussions with experts involved in the topic within ASP were conducted from the perspective of users. Feedback has been collected from users who are interested in using low latency NASA data within application communities of agriculture, disasters, water resources, health and air quality, ecological conservation, wildland fires and capacity building. Analysis-ready NRT data products in a variety of formats have been mentioned many times in the collected feedback, especially for applied users with little to no experience using research-grade earth observation products. Users prefer to have products that can be easily integrated into their existing workflows and take their analysis to the data. HDF5 is a commonly used data format for research, but typically requires some conversion to a more friendly format for applications and regular use in decision-making. Users prefer the GeoTIFF data format that can be directly ingested into a GIS mapping software and platform for data analysis and visualization. For example, LANCE’s fire, flood, SO2 and Black Marble Nighttime Blue/Yellow Composite data products have been integrated into NASA Disasters Mapping Portal, which is an GIS-based open data portal, for users in the disaster management community. There are 291 LANCE NRT layers available through GIBS and Worldview, where users can download a snapshot in GeoTIFF format. Operational users expect data to be processed as close to the user as possible. The collected feedback indicates that LANCE fire products within 3 hours latency would meet the needs of the wildland fire community. The ideal latency for volcanic application is 10-15 minutes. Users in Volcanic Ash Advisory Centers (VAAC) reported that the first forecast volcanic product should be issued within 75 minutes from the volcano eruption [5]. Overall, for disaster applications, data latency within 3 hours is useful while latency greater than 12 hours is not timely enough for operational use. Capacity building and training are critical for users to be able to access, interpret and use data products and tools for their decision making, especially for applied users with limited experience using earth observation products. LANCE data products have been used in a number of capacity building projects domestically and internationally [6]. As LANCE continues to bring new products into the system, users request training to utilize LANCE new and upcoming data products and capabilities in their applications. Due to the limitation of bandwidth and downstream flow paths, users in some developing countries need tools to select and download data for a specific area of interest instead of bulk downloads. The collected feedback also shows the lack of available SAR satellite low latency data products. The advantages of SAR to monitor conditions and changes on the ground through darkness, clouds, volcanic ash, and other atmospheric conditions, are appealing to low latency users. For example, terabytes of low latency but cloudy optical images are not helpful in rapidly identifying the extent of flood or fire impacts. LANCE could be complemented with low latency measurements via the upcoming NASA-ISRO Synthetic Aperture Radar (NISAR) mission [7]. Requests for higher spatial resolution products are expressed. A user from the wildland fire management community reported that products with 30-m spatial resolution could be used to detect small fires. The 30-m Landsat OLI fire data is now part of NASA’s Fire Information for Resource Management System (FIRMS) US/Canada [8]. Within the open and free NASA resources, LANCE disseminates NRT data products in a manner that allows them to be accessible and understandable to both scientific and applied users. In many application areas, latency plays an important or even decisive role where low latency earth observations help people to observe areas of interest, detect and track changes in the environment and make timely decisions. NASA’s Earth Applied Sciences Program promotes the use of LANCE NRT products and builds a bridge between application users and research teams. The collected feedback indicates data latency within 3 hours is useful for most of the applications, and shows the needs of user-friendly, analysis-ready products, and requests training on LANCE’s new and upcoming data products. User feedback has been provided to LANCE UWG for guidance and recommendations, and for translating findings into something actionable.

Tian Yao

Multidimensional Data Aggregation in the Cloud with Application to Geostationary Satellite-based Air Quality Monitoring

Scientists use satellite data for studying Earth's systems, and the remote sensing data that these satellites collect are typically separated into files of a size small enough for efficient network transfer and storage. However, researchers usually prefer to analyze the data based on real-world dimensions like time, space, or elevation. To help with this, NASA's Atmospheric Science Data Center (ASDC) developed a new cloud-based tool that combines these smaller data chunks into larger, more useful datasets. The tool works on Network Common Data Form (netCDF4) and some HDF5 formatted files, and it is available as a service in NASA's Earthdata Cloud. In this presentation, we showcase this service using data from the Tropospheric Emissions: Monitoring of Pollution (TEMPO) instrument. By combining TEMPO's continuous observations over time, we create longer and more informative analysis-ready time series to facilitate the study of air quality patterns. Insights gained will provide a more comprehensive understanding of pollution sources, transport patterns, and their effects on the environment and human health.

Daniel Kaufman

Fostering Geothermal Machine Learning Success: Elevating Big Data Accessibility and Automated Data Standardization in the Geothermal Data Repository

The Department of Energy's (DOE's) Geothermal Data Repository (GDR) has implemented improvements to both its data lakes and its data standards and automated data pipelines. The GDR data lakes have reduced storage and compute-related barriers to using large geothermal datasets, enabling these large datasets to be accessed by anyone with a modern computer and internet access. More recently, the GDR has been working to further reduce barriers through streamlining the data intake process, educating users on the process and requirements, and helping users access data from the data lakes. These improvements have augmented the quantity of datasets the GDR is able to accept into its data lakes and have enabled users who are new to cloud tools to access these datasets more easily, overall increasing the accessibility of big geothermal data for use in machine learning and other projects. In addition, the GDR now has built-in data standards and pipelines for drilling data, geospatial data, and distributed acoustic sensing (DAS) data. These standardization efforts aim to enhance the real-world applicability of geothermal machine learning outcomes by improving the quality of training data. Specifically, through standardizing high-value datasets, the GDR is reducing project-specific data curation requirements, thus allowing more time for actual research. By automating this process, the burden of standardization is lifted from the user, ultimately increasing the availability of standardized data. This paper provides an update on recent improvements made to the GDR's data lakes and automated data pipelines, including: (1) streamlining the data lake intake process, (2) better educating users on the process and requirements through a new data lakes page, (3) adding data lake direct access links to GDR data lake submission pages, (4) implementing a DAS data pipeline to convert DAS data uploaded in SEG-Y format to a standardized hierarchical data format v5 (HDF5), (5) extending this pipeline to encompass data in the GDR data lake, (6) adding metadata requirements for geospatial data, (7) making user interface/user experience (UX) enhancements to the data pipelines' documentation pages, and (8) improving the GDR's data standards and pipelines pages to better guide users in ensuring that their data is standardized by the GDR's automated data pipelines. 2024 Geothermal Resources Council. All rights reserved.

accessibility

Hierarchical Data Format for Earth Observing System Data Product Developer's Guide

The "Hierarchical Data Format for Earth Observing System" talk will address the best practices for creating ESDIS data products. The work presented is done in support of Data Product Developers Guide Working Group with mission "to help data product developers make data usable for end users". During the presentation, we will use some examples of NASA data products and show how to modify them to make data more usable.

Data usability