Search NASA⌕ Search

SEARCH · Search NASA

Results for “HDF”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12
All sourcesNASA NTRSDOE OSTI
NASA NTRS2014

Global Precipitation Measurement (GPM) Mission Products and Services at the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC)

On February 27, 2014, the NASA Global Precipitation Measurement (GPM) mission was launched to provide the next-generation global observations of rain and snow (http:pmm.nasa.govGPM). The GPM mission consists of an international network of satellites in which a GPM Core Observatory satellite carries both active and passive microwave instruments to measure precipitation and serve as a reference standard, to unify precipitation measurements from a constellation of other research and operational satellites. The NASA Goddard Earth Sciences (GES) Data and Information Services Center (DISC) hosts and distributes GPM data within the NASA Earth Observation System Data Information System (EOSDIS). The GES DISC is home to the data archive for the GPM predecessor, the Tropical Rainfall Measuring Mission (TRMM). Over the past 16 years, the GES DISC has served the scientific as well as other communities with TRMM data and user-friendly services. During the GPM era, the GES DISC will continue to provide user-friendly data services and customer support to users around the world. GPM products currently and to-be available include the following:Level-1 GPM Microwave Imager (GMI) and partner radiometer productsLevel-2 Goddard Profiling Algorithm (GPROF) GMI and partner productsLevel-3 daily and monthly productsIntegrated Multi-satellitE Retrievals for GPM (IMERG) products (early, late, and final) A dedicated Web portal (including user guides, etc.) has been developed for GPM data (http:disc.sci.gsfc.nasa.govgpm). Data services that are currently and to-be available include Google-like Mirador (http:mirador.gsfc.nasa.gov) for data search and access; data access through various Web services (e.g., OPeNDAP, GDS, WMS, WCS); conversion into various formats (e.g., netCDF, HDF, KML (for Google Earth), ASCII); exploration, visualization, and statistical online analysis through Giovanni (http:giovanni.gsfc.nasa.gov); generation of value-added products; parameter and spatial subsetting; time aggregation; regridding; data version control and provenance; documentation; science support for proper data usage, FAQ, help desk; monitoring services (e.g. Current Conditions) for applications.

precipitation↗
NASA NTRS2015

Improving the Accessibility and Use of NASA Earth Science Data

Many of the NASA Langley Atmospheric Science Data Center (ASDC) Distributed Active Archive Center (DAAC) multidimensional tropospheric and atmospheric chemistry data products are stored in HDF4, HDF5 or NetCDF format, which traditionally have been difficult to analyze and visualize with geospatial tools. With the rising demand from the diverse end‐user communities for geospatial tools to handle multidimensional products, several applications, such as ArcGIS, have refined their software. Many geospatial applications now have new functionalities that enable the end user to: Store, serve, and perform analysis on each individual variable, its time dimension, and vertical dimension. Use NetCDF, GRIB, and HDF raster data formats across applications directly. Publish output within REST image services or WMS for time and space enabled web application development. During this webinar, participants will learn how to leverage geospatial applications such as ArcGIS, OPeNDAP and ncWMS in the production of Earth science information, and in increasing data accessibility and usability.

Tisdale, Matthew↗
NASA NTRS2016

NASA Briefing for Unidata

The NASA representative to the Unidata Strategic Committee presented a semiannual update on NASAs work with and use of Unidata technologies. The talk covered the program of cloud computing prototypes being undertaken for the Earth Observing System Data and Information System (EOSDIS). Also discussed were dataset interoperability recommendations ratified via the EOSDIS Standards Office and the HDF Product Designer tool with respect to its possible applicability to data in network Common Data Form (NetCDF) version 4.

data integration↗
NASA NTRS2017

Moving from HDF4 to HDF5/netCFD-4

In this presentation, we will go over the major differences between two file formats and libraries, and will talk about the HDF5 features that users should consider when designing new products in HDF5netCDF4. We will also discuss the h4h5tools toolkit that can facilitate conversion of data in the existing HDF4 files to HDF5 and netCDF-4, and we will engage the participants in the discussion of how The HDF Group can help with the transition and adoption of HDF5 and netCDF-4.

netCDF-4↗
NASA NTRS2017

Climate Data Guide - Modern Era Retrospective Analysis for Research and Applications, version 2 (MERRA-2)

The Modern-Era Retrospective analysis for Research and Applications, version 2 (MERRA-2) is a global atmospheric reanalysis produced by the NASA Global Modeling and Assimilation Office (GMAO). It spans the satellite observing era from 1980 to the present. The goals of MERRA-2 are to provide a regularly-gridded, homogeneous record of the global atmosphere, and to incorporate additional aspects of the climate system including trace gas constituents (stratospheric ozone), and improved land surface representation, and cryospheric processes. MERRA-2 is also the first satellite-era global reanalysis to assimilate space-based observations of aerosols and represent their interactions with other physical processes in the climate system. The inclusion of these additional components are consistent with the overall objectives of an Integrated Earth System Analysis (IESA). MERRA-2 is intended to replace the original MERRA product, and reflects recent advances in atmospheric modeling and data assimilation. Modern hyperspectral radiance and microwave observations, along with GPS-Radio Occultation and NASA ozone datasets are now assimilated in MERRA-2. Much of the structure of the data files remains the same in MERRA-2. While the original MERRA data format was HDF-EOS, the MERRA-2 supplied binary data format is now NetCDF4 (with lossy compression to save space).

MERRA-2↗
NASA NTRS2017

Satellite Level 3 & 4 Data Subsetting at NASA GES DISC

Earth Science data are available in many file formats (NetCDF, HDF, GRB, etc.) and in a wide range of sizes, from kilobytes to gigabytes. These properties have become a challenge to users if they are not familiar with these formats or only want a small region of interest (ROI) from a specific dataset. At NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), we have developed and implemented a multipurpose subset service to ease user access to Earth Science data. Our Level 3 & 4 Regridder is capable of subsetting across multiple parameters (spatially, temporally, by level, and by variable) as well as having additional beneficial features (temporal means, regridding to target grids, and file conversion to other data formats). In this presentation, we will demonstrate how users can use this service to better access only the data they need in the form they require.

subsetting↗
NASA NTRS2018

GMI-IPS: Python Processing Software for Aircraft Campaigns

NASA's Atmospheric Tomography Mission (ATom) seeks to understand the impact of anthropogenic air pollution on gases in the Earth's atmosphere. Four flight campaigns are being deployed on a seasonal basis to establish a continuous global-scale data set intended to improve the representation of chemically reactive gases in global atmospheric chemistry models. The Global Modeling Initiative (GMI), is creating chemical transport simulations on a global scale for each of the ATom flight campaigns. To meet the computational demands required to translate the GMI simulation data to grids associated with the flights from the ATom campaigns, the GMI ICARTT Processing Software (GMI-IPS) has been developed and is providing key functionality for data processing and analysis in this ongoing effort. The GMI-IPS is written in Python and provides computational kernels for data interpolation and visualization tasks on GMI simulation data. A key feature of the GMI-IPS, is its ability to read ICARTT files, a text-based file format for airborne instrument data, and extract the required flight information that defines regional and temporal grid parameters associated with an ATom flight. Perhaps most importantly, the GMI-IPS creates ICARTT files containing GMI simulated data, which are used in collaboration with ATom instrument teams and other modeling groups. The initial main task of the GMI-IPS is to interpolate GMI model data to the finer temporal resolution (1-10 seconds) of a given flight. The model data includes basic fields such as temperature and pressure, but the main focus of this effort is to provide species concentrations of chemical gases for ATom flights. The software, which uses parallel computation techniques for data intensive tasks, linearly interpolates each of the model fields to the time resolution of the flight. The temporally interpolated data is then saved to disk, and is used to create additional derived quantities. In order to translate the GMI model data to the spatial grid of the flight path as defined by the pressure, latitude, and longitude points at each flight time record, a weighted average is then calculated from the nearest neighbors in two dimensions (latitude, longitude). Using SciPya's Regular Grid Interpolator, interpolation functions are generated for the GMI model grid and the calculated weighted averages. The flight path points are then extracted from the ATom ICARTT instrument file, and are sent to the multi-dimensional interpolating functions to generate GMI field quantities along the spatial path of the flight. The interpolated field quantities are then written to a ICARTT data file, which is stored for further manipulation. The GMI-IPS is aware of a generic ATom ICARTT header format, containing basic information for all flight campaigns. The GMI-IPS includes logic to edit metadata for the derived field quantities, as well as modify the generic header data such as processing dates and associated instrument files. The ICARTT interpolated data is then appended to the modified header data, and the ICARTT processing is complete for the given flight and ready for collaboration. The output ICARTT data adheres to the ICARTT file format standards V1.1. The visualization component of the GMI-IPS uses Matplotlib extensively and has several functions ranging in complexity. First, it creates a model background curtain for the flight (time versus model eta levels) with the interpolated flight data superimposed on the curtain. Secondly, it creates a time-series plot of the interpolated flight data. Lastly, the visualization component creates averaged 2D model slices (longitude versus latitude) with overlaid flight track circles at key pressure levels. The GMI-IPS consists of a handful of classes and supporting functionality that have been generalized to be compatible with any ICARTT file that adheres to the base class definition. The base class represents a generic ICARTT entry, only defining a single time entry and 3D spatial positioning parameters. Other classes inherit from this base class; several classes for input ICARTT instrument files, which contain the necessary flight positioning information as a basis for data processing, as well as other classes for output ICARTT files, which contain the interpolated model data. Utility classes provide functionality for routine procedures such as: comparing field names among ICARTT files, reading ICARTT entries from a data file and storing them in data structures, and returning a reduced spatial grid based on a collection of ICARTT entries. Although the GMI-IPS is compatible with GMI model data, it can be adapted with reasonable effort for any simulation that creates Hierarchical Data Format (HDF) files. The same can be said of its adaptability to ICARTT files outside of the context of the ATom mission. The GMI-IPS contains just under 30,000 lines of code, eight classes, and a dozen drivers and utility programs. It is maintained with GIT source code management and has been used to deliver processed GMI model data for the ATom campaigns that have taken place to date.

Damon, M. R.↗
NASA NTRS2017

Reusing JPSS Ground System Components to Process Aura Ozone Monitoring Instrument Science Products

New Earth observation instruments are planned to enable advancements in Earth science research over the next decade. Diversity of Earth observing instruments and their observing platforms will continue to increase as new instrument technologies emerge and are deployed as part of National programs such as Joint Polar Satellite System (JPSS), Geostationary Operational Environmental Satellite system (GOES), Landsat as well as the potential for many CubeSat and aircraft missions. The practical use and value of these observational data often extends well beyond their original purpose. The practicing community needs intuitive and standardized tools to enable quick unfettered development of tailored products for specific applications and decision support systems. However, the associated data processing system can take years to develop and requires inherent knowledge and the ability to integrate increasingly diverse data types from multiple sources. This paper describes the adaptation of a large-scale data processing system built for supporting JPSS algorithm calibration and validation (CalVal) node to a simplified science data system for rapid application. The new configurable data system reuses scalable JAVA technologies built for the JPSS Government Resource for Algorithm Verification, Independent Test, and Evaluation (GRAVITE) system to run within a laptop environment and support product generation and data processing of AURA Ozone Monitoring Instrument (OMI) science products. Of particular interest are the root requirements necessary for integrating experimental algorithms and Hierarchical Data Format (HDF) data access libraries into a science data production system. This study demonstrates the ability to reuse existing Ground System technologies to support future missions with minimal changes.

Science Data Systems↗
NASA NTRS2016

NASA WorldWind: Open Source Visualization Technology for Earth Observation

NASA WorldWind: Open Source Visualization Technology for Earth Observation WorldWind, open source virtual globe technology for Java, iOS, Android and Web, is provided by NASA and is architected as API-centric modular componentry. This enable it to be continually optimized and feature-enriched in ways that allow applications based on this SDK (Software Development Kit) to benefit Earth Observation, especially Open Science, with minimal or no adjustment for the decade ahead. The next-generation National Airspace System (NAS) aviation management system for the U.S. Federal Aviation Administration, FAA, uses WorldWind, as do applications currently being developed by the European Space Agency, along with several other US and European government agencies and industry partners. This presentation will demonstrate several NASA open source use cases for WorldWind technology that include advances being made to optimize access to NetCDF and HDF data via WebWorldWind.NASA WorldWind: Multidimensional Geospatial Web Platform The ability to see spatial data in its native context is essential for that data to be appreciated whether by the scientific community, policy and decision-makers or the general public. Recently, the accessibility of spatial data has dramatically improved. Without the need to install an application, spatial data can now be experienced via any web browser, mobile devices included. For developers, by simply updating the app on your server, the latest version of your application is now immediately available to your entire usercommunity. Unlike other virtual globes such as Google Earth, NASA World Wind offers something very special, full control to customize the interface with any features or functionalities you might need. You decide how the data is accessed and experienced. This allows you to provide maximum value of the information to your user community. The web version of NASA WorldWind (WebWorldWind) has made it possible for a whole new suite of applications for managing and sharing spatial data. Apps built with this web version are ideal for immediate social media type activity and also facilitate delivery of sophisticated data exchange scenarios such as weather and climate research, disaster response, personal navigation, and industrial-strength tracking for transportation, supply chain, aviation and satellites. WebWorldWind is an application component, not an app in itself. It is written in JavaScript and provides the real world geographic context for spatial data and information visualization, using a rich set of shapes and graphic primitives. WebWorldWind also provides platform independence, while accommodating any number of data types. Web WorldWind runs on any platform via a browser, i.e., Internet Explorer, Firefox, Chrome and Safari. Features include, 3D virtual globe, 2D map with multiple projection choices (Mercator, Polar, UPS, Equirectangular), imagery and elevation import, extensible, data retrieval (via REST, WMS, WCS, WFS, Bing, User-Defined), decluttering, measurement, accurate line-ofsight, subsurface visualization, and more.

Open Source Mapping↗
NASA NTRS2018

GMI-IPS: Processing & Visualization Software Used in ATom DC-8 Aircraft Studies

NASA's Atmospheric Tomography Mission (ATom) deployed in each of the four seasons during 2016-2018, the DC-8 aircraft in order to establish global-scale datasets intended to improve the representation of chemically reactive gases in global atmospheric chemistry models (ACMs). The Global Modeling Initiative (GMI) executed simulations for each ATom flight using the GMI Chemistry Transport Model (GMI-CTM) to provide species concentrations of chemical gases along the DC-8 flight transects. To solve the problem of translating the GMI-CTM simulation data to the unique spatial resolutions of each ATom flight, the GMI ICARTT Processing Software (GMI-IPS) was developed.The GMI-IPS is written in Python and provides data processing, flight extraction, and visualization support for aircraft research projects using ICARTT format, which is a standard format for airborne instrument data. Additionally, the GMI-IPS interpolates global gridded model data from Hierarchical Data Format (HDF) to ICARTT compatible flight transects. Software classes for instruments and collections provided by the ATom DC-8 aircraft such as MER10, MMS, etc. are derived from a common base class. Other functionality provided by the GMI-IPS are: deriving missing flight entries along a transect, reading ICARTT entries from file, and providing Python data structures for storing flight and model information, and more.The GMI-IPS is GIT source controlled, has approximately 30,000 lines of code, and supports parallelization across data collections. It delivered GMI-CTM data for more than forty distinct DC-8 aircraft flights that took place under ATom. The output ICARTT files adhere to format standard V1.1, and pass the scan utility provided by NASA LaRC Airborne Science Data for Atmospheric Composition. This presentation will include a software and methods overview, and results from ATom, including assessments using the GMI-CTM showing how well observations from ATom flight transects represent a broader region.

Damon, M. R.↗
NASA NTRS2019

Leveraging the Cloud for HDF1 Software Testing

In this talk we will discuss how we leverage the Cloud for HDF software daily regression testing including testing of the HDF5 parallel library on the Cloud cluster using Orange FS.

CI testing↗
NASA NTRS2019

Diamondites: Evidence for a Distinct Tectono-Thermal Diamond-Forming Event Beneath the Kaapvaal Craton

The petrogenesis and relationship of diamondite to well-studied monocrystalline and fibrous diamonds are poorly understood yet would potentially reveal new aspects of how diamond-forming fluids are transported through the lithosphere and equilibrate with surrounding silicates. Of twenty-two silicate- and oxide-bearing diamondites investigated, most yielded garnet intergrowths (n = 15) with major element geochemistry (i.e. Ca-Cr) classifying these samples as low-Ca websteritic or eclogitic. The garnet REE patterns fit an equilibrium model suggesting the diamond-forming fluid shares an affinity with high-density fluids (HDF) observed in fibrous diamonds, specifically on the join between the saline–carbonate end- members. The δ13C values for the diamonds range from -5.27 to -22.48 ‰ (V-PDB) with δ18O values for websteritic garnets ranging from +7.6 to +5.9 ‰ (V-SMOW). The combined C-O stable isotope data support a model for a hydrothermally altered and organic carbon-bearing subducted crustal source(s) for the diamond- and garnet-forming media. The nitrogen aggregation states of the diamonds require that diamondite-formation event(s) pre-dates fibrous diamond-formation and post-dates most of the gem monocrystalline diamond-formation events at Orapa. The modelled fluid compositions responsible for the precipitation of diamondites match the fluid-poor and fluid-rich (fibrous) monocrystalline diamonds, where all grow from HDFs within the saline-silicic-carbonatitic ternary system. However, while the nature of the parental fluid(s) share a common lithophile element geochemical affinity, the origin(s) of the saline, silicic, and/or carbonatitic components of these HDFs do not always share a common origin. Therefore, it is wholly conceivable that the diamondites are evidence of a distinct and temporally unconstrained tectono-thermal diamond-forming event beneath the Kaapvaal craton.

S Mikhail↗
NASA NTRSDate not supplied

Cloud Optimized Data Formats

Cloud computing offers the promise of being able to analyze Big Data earth Observations at scale, by allowing scientists to deploy many nodes at once to analyze the data. However, in order to take full advantage of cloud scalability, it is often necessary to reorganize and reformat the data to enable fine-grained, parallel access to the data in Web Object Storage. NASA recently conducted a study of several formats that are optimized for analysis in the cloud: Parquet, zarr, HDF (Hierarchical Data Format) in the Cloud, and Cloud-Optimized GeoTIFF (Tagged Image File Format). They were compared against non-cloud-optimized formats, netCDF (network Common Data Form) and GeoTIFF, with criteria based both on stewardship and analysis performance.

Christopher Lynnes↗
NASA NTRS2021

Southern Idaho Health & Air Quality: Monitoring Atmospheric Mixing Heights Post-Wildfire Through the Use of NASA Earth Observations

Wildfire smoke has long-lasting impacts on public and environmental health. Currently, agencies that monitor smoke base their decisions on an analysis of how fires burn, the direction the smoke moves from the fire source, and unverified estimates of mixing height. Mixing heights describe the maximum altitude to which a smoke plume rises. Satellite imagery provides more continuous and accurate coverage of mixing heights than current in situ methods. Thus, the team developed a software tool that processes and extracts mixing height observations from Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations (CALIPSO) Vertical Feature Mask granules. The team partnered with the National Oceanic and Atmospheric Administration’s National Weather Service, the Bureau of Land Management’s National Interagency Fire Center, and the National Park Service’s Fire Management Program Center to analyze historic fire events in southern Idaho. To do so, they used Suomi National Polar-orbiting Partnership (Suomi NPP) Visible Infrared Imaging Radiometer Suite (VIIRS), and Terra and Aqua Moderate Resolution Imaging Spectroradiometer (MODIS) to verify where a CALIPSO pass intersects thermal anomalies and smoke plumes. The software extracts features of relevance from the hdf file of each CALIPSO transect to locate layers of continuous aerosols. The maximum altitude at which the aerosol ends is recorded as the mixing height, along with a matching latitude and longitude. The satellite-derived values can be used to validate past mixing height predictions and evaluate the accuracy and systematic bias of different estimation methods. These results may allow agencies to make better comparisons and subsequent smoke pollution management, prevention, and public health decisions if the spatial and temporal differences between predictions and observations can be resolved.

DEVELOP Project Summary↗
NASA NTRSDate not supplied

Making NASA GES DISC Level 2 Data GIS Analysis Ready

There are many valuable data hosted by NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) for GIS applications in the areas of extreme weather events, climatic anomaly, and public health. However, using NASA Earth Science Data poses some challenges for GIS users. Many of these users are not experts in Earth Observation and have little knowledge about NASA's Earth science data. In the GIS community, GeoTiff is the most widely used raster format, whereas NASA's data is primarily in complex multidimensional netCDF and HDF formats. This complexity makes it difficult for GIS users, especially those who are unfamiliar with these formats. Although GIS software like ArcGIS has made progress in processing multidimensional netCDF data, certain issues still remain, particularly with level 2 data. In this study, we use TROPSpheric Monitoring instrument (TROPOMI) level 2 data as an example to demonstrate how to make such data GIS analysis ready. The process involves: 1. Creating a feature layer from the TROPOMI level 2 data. 2. Converting the feature layer to a gridded raster dataset. 3. Mosaicking gridded raster datasets into a raster dataset covering the entire desired extent. 4. Generating a symbology with a GIBS-specific style that aligns with the visual standards and requirements of GIBS. 5. Publishing image services. By performing these preprocessing and transformation steps, NASA level 2 data can be made compatible and ready for use within GIS software for various spatial analysis and visualization tasks.

Geographic Information System↗
NASA NTRSDate not supplied

ASDC’s Python-Based Metadata Extraction Pipeline for Suborbital Campaigns

The FAIRness of data products, especially findability and accessibility depend on rich metadata which, when extracted, can allow for proper curation. Over the past few years, the Atmospheric Science Data Center (ASDC) suborbital science support team has developed a metadata extraction pipeline to ensure the required metadata can be retrieved systematically, effectively, and efficiently to ensure the data can be used by a broad community. The development of a pipeline has presented many, but necessary, challenges to support archival and distribution of ASDC’s 30+ suborbital missions. Though sufficient metadata is provided by instrument scientists, the metadata may not be readily machine actionable due to different formats and templates. Further complicating metadata extraction, our team has found that the nature of metadata can be quite diverse given the difference in measurement types, instruments, and measurement platforms. A metadata extraction pipeline has been developed to provide an efficient, plugin-in based, method for adding new parsers, a configuration system that lets non-developers customize how files are processed, and a system for identifying and logging metadata quality issues to ensure they are readily found and addressed. The metadata extraction pipeline identifies critical pieces of metadata that are needed to promote data FAIRness, including location, file revision, measurement start/end datetime and can be easily modified to extract further information (such as variables). Given the wide-ranging datasets, the pipeline has been modified to accommodate multiple file formats, including multiple versions of ICARTT (International Consortium for Atmospheric Research on Transport and Transformation), HDF (Hierarchical Data Format), netCDF (network Common Data Form), and multiple versions of the Ames File Format. The pipeline also supports building metadata for file formats that cannot have metadata easily extracted from them, such as PDF (Portable Document Format) and GIF (Graphics Interchange Format). The pipeline has allowed our team to maintain a consistent flow of data and metadata to archival and distribution services, ensuring the ASDC meets the needs of the suborbital science community. This presentation will highlight the ASDC’s suborbital metadata extraction pipeline, its development, how it’s been modified to support data FAIRness, and plans for maintaining the pipeline and adding new features.

Abraham Porter↗