Cloud-Optimized Data Access using OPeNDAP’s Dataset Metadata Response (DMR++)
Explore the source record for details and available documents.
Patrick Michael Quinn↗
SEARCH · Search NASA
Results for “datasets metadata”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Enabling FAIR data in Earth and environmental science with community-centric (meta)data reporting formats
Abstract Research can be more transparent and collaborative by using Findable, Accessible, Interoperable, and Reusable (FAIR) principles to publish Earth and environmental science data. Reporting formats—instructions, templates, and tools for consistently formatting data within a discipline—can help make data more accessible and reusable. However, the immense diversity of data types across Earth science disciplines makes development and adoption challenging. Here, we describe 11 community reporting formats for a diverse set of Earth science (meta)data including cross-domain metadata (dataset metadata, location metadata, sample metadata), file-formatting guidelines (file-level metadata, CSV files, terrestrial model data archiving), and domain-specific reporting formats for some biological, geochemical, and hydrological data (amplicon abundance tables, leaf-level gas exchange, soil respiration, water and sediment chemistry, sensor-based hydrologic measurements). More broadly, we provide guidelines that communities can use to create new (meta)data formats that integrate with their scientific workflows. Such reporting formats have the potential to accelerate scientific discovery and predictions by making it easier for data contributors to provide (meta)data that are more interoperable and reusable.
54 ENVIRONMENTAL SCIENCES↗
ESS-DIVE Reporting Format for Dataset Package Metadata
ESS-DIVE’s (Environmental Systems Science Data Infrastructure for a Virtual Ecosystem) dataset metadata reporting format is intended to compile information about a dataset (e.g., title, description, funding sources) that can enable reuse of data submitted to the ESS-DIVE data repository. The files contained in this dataset include instructions (dataset_metadata_guide.md and README.md) that can be used to understand the types of metadata ESS-DIVE collects. The data dictionary (dd.csv) follows ESS-DIVE’s file-level metadata reporting format and includes brief descriptions about each element of the dataset metadata reporting format. This dataset also includes a terminology crosswalk (dataset_metadata_crosswalk.csv) that shows how ESS-DIVE’s metadata reporting format maps onto other existing metadata standards and reporting formats.Data contributors to ESS-DIVE can provide this metadata by manual entry using a web form or programmatically via ESS-DIVE’s API (Application Programming Interface). A metadata template (dataset_metadata_template.docx or dataset_metadata_template.pdf) can be used to collaboratively compile metadata before providing it to ESS-DIVE.Since being incorporated into ESS-DIVE’s data submission user interface, ESS-DIVE’s dataset metadata reporting format, has enabled features like automated metadata quality checks, and dissemination of ESS-DIVE datasets onto other data platforms including Google Dataset Search and DataCite.
54 ENVIRONMENTAL SCIENCES↗
Search Enhancements using Natural Language Processing Techniques
NASA Goddard Earth Sciences Data and Information Services Center (GESDISC) is one of the 12 NASA Science Mission Directorate Data Centers. The main goal of GESDISC is to provide earth science data, information, and services to the earth science data community. Consequently, data discovery is at the center of our mission and our search engine is the primary tool for our users to interact, find, and access our data. Existing search approaches are largely focused on hard-matching of keywords in the search query with dataset metadata. Here we propose to expand the search by introducing a complementary natural language processing (NLP) search. At the heart of our proposed NLP search, we trained a joint embedding using scientific text corpus and a curated set of dataset metadata. The embedding learns the association between words in our dataset metadata and those of the scientific text corpus. This enables us to go beyond simple hard-matching of a query and data set metadata and have a notion of “similarity” between the search query and the datasets. We further integrated our NLP search into the Elastic Search (ES) framework leveraging similarity search capabilities offered through the “dense_vector” field type. Our preliminary evaluations show that our proposed NLP search has the potential to be utilized to complement the existing search engine and serve as a base for a dataset recommendation system.
Armin Mehrabian↗
Improving GES Disc Data Search and Discovery Through AI Metadata Augmentation
NASA’s Goddard Earth Science (GES) Data and Information Services Center (DISC) is one of twelve data centers in NASA's Science Mission Directorate (SMD), providing vital earth science data to a diverse user base. To enhance the discoverability of this data, GES DISC employs a keyword search system, which leverages scientific keywords embedded in dataset metadata. However, the evolving nature of scientific applications of our data necessitates regular review and augmentation of these keywords. To address this, we developed a service to automatically predict missing science keywords in the metadata. This service constructs a knowledge graph from the latest GES DISC metadata within NASA’s Common Metadata Repository (CMR). Using an open-source library, we trained a machine learning model to predict absent science keywords in the metadata. Our preliminary results indicate that the model has high levels of accuracy at predicting science keywords in the dataset metadata when exposed to data not included in its training. These predicted keywords were then evaluated by GES DISC data curation scientists and compared against other AI tools for metadata augmentation. We aim to enhance the overall usability and accessibility of NASA’s earth science data by implementing this tool in our data curation processes.
Kendall Gilbert↗
Application of a Dataset-Publication Knowledge Graph for Improving Earth Science Data Search
Finding a dataset at a NASA data center that is the best fit for the researcher’s application presents a challenge, not only for a novice user but for an experienced one, due to the data complexity and a multitude of choices of the existing data. Users often search for the data based on the application they are interested in, their research domain, phenomena, research topic, etc. As existing dataset metadata may not cover these search terms, the user may not obtain the most relevant results for their purpose. This problem was addressed by leveraging the content of the titles and abstracts of the research papers that utilize NASA datasets. For this, features from the paper titles and abstracts were extracted, and then a knowledge graph (KG) was used to link these features to the datasets used in that paper. The search for the datasets was tested by querying this knowledge graph through various terms extracted from Earth Science ontologies such as Semantic Web for Earth and Environment Technology (SWEET), and it was shown that this KG search outperforms the existing search that exclusively queries the dataset metadata.
Kristina Stoyanova↗
Report on the Global Data Assembly Center (GDAC) to the 12th GHRSST Science Team Meeting
In 2010/2011 the Global Data Assembly Center (GDAC) at NASA's Physical Oceanography Distributed Active Archive Center (PO.DAAC) continued its role as the primary clearinghouse and access node for operational Group for High Resolution Sea Surface Temperature (GHRSST) datastreams, as well as its collaborative role with the NOAA Long Term Stewardship and Reanalysis Facility (LTSRF) for archiving. Here we report on our data management activities and infrastructure improvements since the last science team meeting in June 2010.These include the implementation of all GHRSST datastreams in the new PO.DAAC Data Management and Archive System (DMAS) for more reliable and timely data access. GHRSST dataset metadata are now stored in a new database that has made the maintenance and quality improvement of metadata fields more straightforward. A content management system for a revised suite of PO.DAAC web pages allows dynamic access to a subset of these metadata fields for enhanced dataset description as well as discovery through a faceted search mechanism from the perspective of the user. From the discovery and metadata standpoint the GDAC has also implemented the NASA version of the OpenSearch protocol for searching for GHRSST granules and developed a web service to generate ISO 19115-2 compliant metadata records. Furthermore, the GDAC has continued to implement a new suite of tools and services for GHRSST datastreams including a Level 2 subsetter known as Dataminer, a revised POET Level 3/4 subsetter and visualization tool, a Google Earth interface to selected daily global Level 2 and Level 4 data, and experimented with a THREDDS catalog of GHRSST data collections. Finally we will summarize the expanding user and data statistics, and other metrics that we have collected over the last year demonstrating the broad user community and applications that the GHRSST project continues to serve via the GDAC distribution mechanisms. This report also serves by extension to summarize the activities of the GHRSST Data Assembly and Systems Technical Advisory Group (DAS-TAG).
sea surface temperature (SST)↗
Screening Tool for Equitable Adoption and Deployment of Solar (STEADy Solar)
The Screening Tool for Equitable Adoption and DeploYment of Solar (STEADy Solar) is a database and mapping tool designed to promoting clean energy investments for low-income communities across the United States. The tool indicates locations that may be eligible for the Investment Tax Credit bonus adders defined in the 2022 Inflation Reduction Act (IRA) and combines this information with demographics, social vulnerability, solar technical potential, solar economics (modeled net present value), and building counts by use-type. It can be used by states, municipalities, community-based organizations, developers, and researchers to identify sites where solar projects may be economical and where federal incentives may be available to support equitable adoption of solar. Specific values include: Areas eligible for the Energy Communities Tax Credit Bonus Program (including brownfield site counts) Areas eligible for the Low Income Communities Bonus Credit Program (including Tribal Lands, and covered affordable housing project counts) Areas categorized as disadvantaged by Justice40 Commercial and Residential Solar economics characterized by the Net Present Value and Simple Payback Period Total Population, Race, and Ethnicity Median Household Income, Poverty rate, Household Tenure Social Vulnerability Count of buildings, developable rooftop solar capacity (in kWdc) and estimated annual generation potential (in kWh) on four building types: Government General Services, Government Emergency Response, Grade Schools, and Colleges/Universities. The linked report describes the STEADy dataset metadata and presents high level insights from the data. The downloadable and formatted excel dataset makes it easy for users to gain insights for their locations. Supporting .csv and shapefiles provide users with the full data to run their own analyses on equitable solar siting.
14 SOLAR ENERGY↗
VizBrick: A GUI-based Interactive Tool for Authoring Semantic Metadata for Building Datasets
Brick ontology is a unified semantic metadata schema to address the stand-ardization problem of buildings' physical, logical, and virtual assets and the relationships between them. Creating a Brick model for a building dataset means that the dataset's contents are semantically described using the standard terms defined in the Brick ontology. It will enable the benefits of data standardization, without having to recollect or reorganize the data and opens the possibility of automation leveraging the machine readability of the semantic metadata. The problem is that authoring Brick models for building datasets often requires knowledge of semantic technology (e.g., on-tology declarations and RDF syntax) and leads to repeated manual trial and error processes, which can be time-consuming and challenging to do with-out an interactive visual representation of the data. We developed VizBrick, a tool with a graphical user interface that can assist users in creating Brick models visually and interactively without having to understand the Re-source Description Framework (RDF) syntax. VizBrick provides handy ca-pabilities such as keyword search for easy find of relevant brick concepts and relations to their data columns and automatic suggestions of concept mapping. In this demonstration, we present a use-case of VizBrick to show-case how a Brick model can be created for a real-world building dataset.
Lee, Sangkeun (Matt)↗
CROCUS Urban Fluxes of CO₂, H₂O, and Turbulence at University of Illinois Chicago
As of May 12, 2026 this dataset is currently being versioned to include data up to April 2026. Once the versioning process is complete, new data files will be available for access. The dataset metadata will also be updated to reflect the data availability of the new data being versioned. This dataset was collected at the UIC Plant Research Laboratory in Chicago, Illinois, as part of the Community Research on Climate and Urban Science (CROCUS) Urban Integrated Field Laboratory (UIFL) project, led by Argonne National Laboratory. The site provides continuous atmospheric flux measurements, focusing on CO₂, H₂O, and heat and momentum transport in an urban setting. The data is processed at 30 minutes interval using the Eddy Covariance method and includes quality control and diagnostic data generated by EddyPro software. The data is stored in the netCDF files following CF conventions. The UIC Plant Research Laboratory is located near major highways and urban infrastructure, including buildings and parking areas. The surrounding landscape consists of a mix of turf, plants, trees, and impervious surfaces such as concrete and asphalt, making it ideal for studying urban at for studies on urban sustainability, air quality, and the effects of urbanization on atmospheric processes on urban climate dynamics, air quality, and surface-atmosphere exchanges within the city of Chicago. This dataset is funded by the U.S. Department of Energy’s Office of Science, Biological and Environmental Research (BER) program.
54 ENVIRONMENTAL SCIENCES↗
Heuristics for Relevancy Ranking of Earth Dataset Search Results
As the Variety of Earth science datasets increases, science researchers find it more challenging to discover and select the datasets that best fit their needs. The most common way of search providers to address this problem is to rank the datasets returned for a query by their likely relevance to the user. Large web page search engines typically use text matching supplemented with reverse link counts, semantic annotations and user intent modeling. However, this produces uneven results when applied to dataset metadata records simply externalized as a web page. Fortunately, data and search provides have decades of experience in serving data user communities, allowing them to form heuristics that leverage the structure in the metadata together with knowledge about the user community. Some of these heuristics include specific ways of matching the user input to the essential measurements in the dataset and determining overlaps of time range and spatial areas. Heuristics based on the novelty of the datasets can prioritize later, better versions of data over similar predecessors. And knowledge of how different user types and communities use data can be brought to bear in cases where characteristics of the user (discipline, expertise) or their intent (applications, research) can be divined. The Earth Observing System Data and Information System has begun implementing some of these heuristics in the relevancy algorithm of its Common Metadata Repository search engine.
science data management↗
Relevancy Ranking of Satellite Dataset Search Results
As the Variety of Earth science datasets increases, science researchers find it more challenging to discover and select the datasets that best fit their needs. The most common way of search providers to address this problem is to rank the datasets returned for a query by their likely relevance to the user. Large web page search engines typically use text matching supplemented with reverse link counts, semantic annotations and user intent modeling. However, this produces uneven results when applied to dataset metadata records simply externalized as a web page. Fortunately, data and search provides have decades of experience in serving data user communities, allowing them to form heuristics that leverage the structure in the metadata together with knowledge about the user community. Some of these heuristics include specific ways of matching the user input to the essential measurements in the dataset and determining overlaps of time range and spatial areas. Heuristics based on the novelty of the datasets can prioritize later, better versions of data over similar predecessors. And knowledge of how different user types and communities use data can be brought to bear in cases where characteristics of the user (discipline, expertise) or their intent (applications, research) can be divined. The Earth Observing System Data and Information System has begun implementing some of these heuristics in the relevancy algorithm of its Common Metadata Repository search engine.
science data management↗
Development of a Knowledge Graph for Dataset Discovery and Identification at a NASA Data Center
The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) archives and distributes hundreds of Earth Science data collections to the public. These collections are used in research, resulting in the publication of thousands of scientific papers each year. As new users come to GES DISC for data, it is important for them to understand how prior research used the data. To help researchers, a knowledge graph (KG) was designed and implemented to connect publication citations with dataset metadata. The relationships created in the graph have the potential to allow the Web applications that utilize this information to directly connect the publication to the GES DISC datasets and services. These relationships are demonstrated using a web application prototype. In addition, the graph can also make connections between publications, datasets, and measurements based on the mentions of datasets and their attributes in the publications. To demonstrate this capability, a web application was created that takes the excerpt from the publication and returns a most likely dataset and measurement pairing, ranking the results based on how often these datasets and measurements were used in prior publications.
Nathaniel Crosby↗
CROCUS High-Frequency Measurements of CO₂, H₂O, Wind, and Temperature at University of Illinois Chicago
As of April 9, 2026: this dataset is currently being versioned to include data up to March 31, 2026. Once the versioning process is complete, new data files will be available for access. The dataset metadata will also be updated to reflect the data availability of the new data being versioned. Raw atmospheric measurements collected at the University of Illinois Chicago (UIC) as part of the Community Research on Climate and Urban Science (CROCUS) project. The data includes high-frequency measurements of carbon dioxide (CO₂) concentration, water vapor (H₂O) concentration, wind speed (U, V, W components), temperature, and atmospheric pressure. These raw data are recorded by the LI-7500DS Open Path CO₂/H₂O Analyzer and a sonic anemometer at a 10 Hz acquisition frequency, providing the necessary inputs for calculating fluxes of CO₂, H₂O, heat, and momentum. In addition to gas concentration measurements, the dataset includes diagnostic information from the instruments, including absorptance, sample and reference signals, and diagnostic values. The data were collected to study urban atmospheric conditions and contribute to flux calculations for urban climate research. These measurements form the basis for calculating 30-minute average fluxes of key atmospheric variables using the eddy covariance method. This dataset provides critical raw input data for researchers interested in atmospheric fluxes, urban air quality, and the interaction between the urban environment and atmospheric processes. The data is part of the U.S. Department of Energy’s Biological and Environmental Research (BER) program, under the CROCUS Urban Integrated Field Laboratory (UIFL) project.
54 ENVIRONMENTAL SCIENCES↗
A Practical Application Using ISO Metadata - Incorporating ISO Metadata into SMAP Data Products
No abstract available
series metadata↗
High Performance Access to Archival Data Stored in HDF4 and HDF5 on Cloud Object Stores Without Reformatting the Files
Cloud computing offers numerous advantages for users of extensive Earth science data collections. These benefits encompass direct online access to data files and granules from any location, scalable access supporting parallel computing workflows, and flexible computing tools enabling innovative experimentation with processing techniques. However, older archival file formats designed for distinct computing systems hinder efficient access to decade-long time-series data when compared to data stored in modern cloud-optimized formats like Web Object Stores (WOS), exemplified by Amazon Web Services’ Simple Storage Service (S3). We describe DMR++ (Dataset Metadata Response plus plus), a technology facilitating efficient access to HDF5 (Hierarchical Data Format, version 5) and HDF4 files stored on WOS systems without requiring data reformatting. DMR++ achieves performance comparable to technologies like Zarr while preserving the original file structure, a substantial benefit considering the vast quantity of archival files held by organizations such as NASA. Moreover, DMR++ typically outperforms cloud-optimized versions of HDF5. Essentially an XML (Extensible Markup Language) document usually stored alongside the described data, DMR++ can also be generated on-the-fly but is generally created during data staging to the WOS. Archival files that use HDF4/5 often store large arrays of numerical data. The data in these files is often compressed, typically reducing their size by a factor of four or more. To achieve efficient access to portions of those arrays, they are 'chunked' into smaller sub-arrays, each individually compressed. The chunk size is a compromise, where spinning disks can efficiently access data in smaller chunks while S3 favors larger chunks. A simple optimization of aggregating smaller chunks that are stored adjacently, transferring them in a single access and then individually decompressing them will improve performance. NASA data pose an additional challenge: special Application Programmer Interface (API) libraries are often needed to compute some variables. These libraries are incompatible with WOS environments. Our solution involves storing computed values in the DMR++ document or a companion file, making them accessible like other variables and eliminating the need for specialized APIs. We outline specific optimizations for both satellite grid and swath data stored in HDF4-EOS2 (Earth Observing System).
James Gallagher↗
Avian Activity Classification Using Recurrent Networks to Fuse Videos with Metadata on Imbalanced Datasets
Activity classification plays a crucial role in various real-life scenarios involving both humans and animals. There is an increasing need for precise activity classification focused on avian-solar interactions, as the usage of solar energy facilities, such as photovoltaic array power stations, has been observed to impact bird species richness, behavior, and activity. However, there has been no work to develop an automated system to monitor and classify these avian-solar interactions. All current methods rely on human observers, which is time and human resources costly and subject to errors related to searcher efficiency. With the recent success of Deep Learning models in activity classification problems, this paper develops a recurrent neural network-based model to automatically classify six avian activities around solar energy facilities. Our proposed model integrates critical feature engineering metadata with video frame data, enabling improved learning and more accurate activity classification. Furthermore, we address the challenge of data imbalance during training and demonstrate the efficacy of our model in detecting and classifying different activities within video tracks. Additionally, we analyze the saliency/backpropagation map of the trained proposed model and validate its decision-making rationale.
Avian activity classification; bidirectional LSTM;↗
Comprehensive Database of Environmental Mitigations Extracted from FERC-Licensed Hydropower Projects Using Artificial Intelligence Techniques, 1998-2023
This dataset provides a comprehensive inventory of environmental mitigation measures required by Federal Energy Regulatory Commission (FERC) licensed hydropower facilities from 461 licenses that were issued from 1998 to 2023. These licenses constitute 446 of the 1015 FERC projects that were active at the end of 2023. 17,612 mentions of environmental mitigations were identified and categorized in 128 unique categories. Mitigations were identified using a Natural Language Processing (NLP) approach, specifically with a Bidirectional Encoder Representations from Transformer (BERT) model. Model-derived results were then reviewed and updated by a subject matter expert as needed. This dataset introduces important enhancements to previous efforts to inventory environmental mitigations, such as including associated license text for each mitigation, tracking the number of instances a mitigation was identified within a license, and providing improved location information. These enhancements significantly expand the dataset's utility, offering greater analytical capabilities and ensuring reproducibility. The dataset is downloadable as a zip file containing the metadata and dataset files.
Ruggles, Thomas [Oak Ridge National Laboratory (OR↗