Results for “FAIR dataset” — Page 3

DOE OSTI2021

Closing the Gap between FAIR Data Repositories and Hierarchical Data Formats

Many in the scientific community, particularly in publicly funded research, are pushing to adhere to more accessible data standards to maximize the findability, accessibility, interoperability, and reusability (FAIR) of scientific data, especially with the growing prevalence of machine learning augmented research. Online FAIR data repositories, such as the Open Science Framework (OSF), help facilitate the adoption of these standards by providing frameworks for storage, access, search, APIs, and other features that create organized hubs of scientific data. However, the wider acceptance of such repositories is hindered by the lack of support of hierarchical data formats, such as Technical Data Management Streaming (TDMS) and Hierarchical Data Format 5 (HDF5), that many researchers rely on to organize their datasets. Various tools and strategies should be used to allow hierarchical data formats, FAIR data repositories, and scientific organizations to work more seamlessly together. A pilot project at Los Alamos National Laboratory (LANL) addresses the disconnect between them by integrating the OSF FAIR data repository with hierarchical data renderers, extending support for additional file types in their framework. The multifaceted interactive renderer displays a tree of metadata alongside a table and plot of the data channels in the file. This allows users to quickly and efficiently load large and complex data files directly in the OSF webapp. Users who are browsing files can quickly and intuitively see the files in the way they or their colleagues structured the hierarchical form and immediately grasp their contents. This solution helps bridge the gap between hierarchical data storage techniques and FAIR data repositories, making both of them more viable options for scientific institutions like LANL which have been put off by the lack of integration between them.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

CAMELSH: A Large-Sample Hourly Hydrometeorological Dataset and Attributes at Watershed-Scale for CONUS

We present CAMELSH (Catchment Attributes and Hourly HydroMeteorology for Large-Sample Studies), the first large-sample hydrometeorological dataset at the hourly scale for the contiguous United States. CAMELSH intergrates hourly meteorological time series, catchment attributes and boundaries from GAGES-II and HydroATLAS for 9,008 catchments across diverse climatic, hydrological, and anthropogenic conditions. In addition, hourly streamflow time series is provided for 3,166 catchments. The dataset spans 45 years (1980–2024) with 11 meteorological variables from the NLDAS-2 forcing dataset, from which we compute nine climate indices related to precipitation, evapotranspiration, seasonality, and snow fraction. Additionally, CAMELSH includes two sets of catchment attributes: 439 from GAGES-II and 195 derived from HydroATLAS. These attributes include factors related to climate, geology, hydrology, river/stream morphology, landscape, nutrient, soil, topography, and anthropogenic influences. Developed in accordance with FAIR (Findability, Accessibility, Interoperability, and Reusability) principles, CAMELSH is the first large-sample dataset at an hourly timescale, supporting machine learning applications for short-term streamflow (flood) prediction and advancing data-driven hydrological research across multiple timescales.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2022

Multimodal sensor fusion framework for residential building occupancy detection

For several years now, smart building energy systems have been a research area of intensive activity. In light of the increasing need for sustainable buildings and energy systems, this trend motivates an increasing need for a solution to reduce carbon dioxide emissions and improve energy efficiency. This work proposes a high-performing and transferable occupancy detection framework that combines sensor data from different data modalities, including time series environmental data (temperature, humidity, and illuminance), image data, and acoustic energy data using ensemble method. To draw out the best prediction performance in each modality, the proposed framework was developed, including various models that were designed to learn the occupancy patterns reflected in the physical data streams. To tackle the time series environmental data, we designed two variants of an occupancy detection spatiotemporal pattern network (Occ-STPN) that performs both feature level and decision level fusion, respectively. We also propose a new metric; the fading memory mean square error (FMMSE), that provides a fair evaluation and penalization of delayed occupancy predictions. Multiple open-sourced datasets, including the Electricity Consumption and Occupancy and the University of California, Irvine's (UCI) building occupancy detection dataset, along with our own real data collected from six different houses, were used to validate the algorithms' performance. The experimental results presented herein break down the performance for each sensing modality, and a detailed analysis of the performance is also discussed.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2025

An AI-Enabled Chat Bot for DuraMAT

The DuraMAT Data Hub has evolved to meet the ever expanding research demands by integrating a chatbot interface that enhances FAIR compliance and simplifies discovery across our growing archive of projects and datasets. With the Data Hubs increasing use by both consortium researchers and the international community, traditional navigation methods have become unwieldy. Building on last year's feasibility study, we now report the successful implementation of the chatbot system, detailing its architecture and demonstrating its ability to deliver a more intuitive and rewarding user experience.

14 SOLAR ENERGY↗

NASA NTRS2021

Thunderstorm and Fair-weather Quasi-Static Electric Fields Over Land and Ocean

Natural lightning and the associated clouds are known to behave differently over land and ocean, but many questions remain. We expand the related observational datasets by obtaining simultaneous quasi-static electric field observations over coastal land, near-shore water, and deep ocean regions during both fair-weather and thunderstorm periods. Oceanic observations were obtained using two 3-m NOAA buoys that were instrumented with Campbell Scientific electric field mills to measure the quasi-static electric fields. These data were compared to selected electric field records from the existing on-shore electric field mill suite of 31 sensors at Kennedy Space Center (KSC). Lightning occurrence times, locations and peak current estimates for both onshore and ocean were provided by the U.S. National Lightning Detection Network. The buoy instruments were first evaluated on-shore at the Florida coast, and the first system was calibrated for field enhancements and to confirm proper behavior of the system in elevated-field environments. The buoys were then moored 20 mi and 120 mi off the coast of KSC in February (20 mi) and August (120 mi) 2014. Diurnal fair-weather fields at both ocean sites matched will with each other and with those found during the Carnegie cruise, but mean values were 33% smaller, due at least in-part to constraints on the calibration procedure. Diurnal fair-weather fields variations at coastal and inland sites were a poorer match than offshore, likely because the offshore environment is “cleaner” with limited variations in local space charge, lower surface aerosol densities, little surface heating to disturb the surface charge layer during fair weather, and fewer local radioactive sources to modulate the near-surface electrical conductivity. Storm-related static fields were 4-5× larger at both oceanic sites than over land, likely due to decreased screening by near-surface space charge produced by corona current. The time-evolution of the electric field and field changes during storm approach are sufficiently different over land and ocean to warrant further study. This work shows the quality, accuracy, and reliability of these data, and has demonstrated the practicality of off-shore electric field measurements for safety- and launch-related decision making at KSC.

Lightning↗

DOE OSTI2026

CAD-VAE: Leveraging Correlation-Aware Latents for Comprehensive Fair Disentanglement

While deep generative models have significantly advanced representation learning, they may inherit or amplify biases and fairness issues by encoding sensitive attributes alongside predictive features. Enforcing strict independence in disentanglement is often unrealistic when target and sensitive factors are naturally correlated. To address this challenge, we propose CAD-VAE(Correlation-Aware Disentangled VAE), which introduces a correlated latent code to capture the information shared between the target and sensitive attributes. Given this correlated latent, our method effectively separates over-lapping factors without extra domain knowledge by directly minimizing the conditional mutual information between target and sensitive codes. A relevance-driven optimization strategy refines the correlated code by efficiently capturing essential correlated features and eliminating redundancy. Extensive experiments on benchmark datasets demonstrate that CAD-VAE produces fairer representations, realistic counterfactuals, and improved fairness-aware image editing.

Ma, Chenrui [University of California Irvine]↗

NASA NTRS1993

A satellite-derived climatology of the ITCZ

This paper presents fundamental climatological characteristics of the intertropical convergence zone (ITCZ) in a simple concise manner using the highly reflective cloud (HRC) dataset. This satellite-derived dataset uses both visible and infrared observations to measure the frequency of occurrence of large-scale convective systems over the global tropics at a 1 deg spatial resolution. These dataset characteristics make the HRC particularly well suited for climatological analysis of the ITCZ because the dataset is based on estimates of organized deep convective cloud systems rather than observations of clouds as a whole, and it provides the spatial resolution needed to identify these large-scale convective structures. Furthermore, the dataset covers a time period extending nearly two decades, which provides for a fairly robust climatology and the opportunity to examine seasonal and interannual variability of both the convection and the latitude of the ITCZ.

Waliser, Duane E.↗

NASA NTRSDate not supplied

Off-Nominal Event Analysis in Autonomous Flights Based on Explainable Artificial Intelligence

A key objective in the Urban Air Mobility program at NASA is to intelligently perform an autonomous flight in a complex urban environment under all weather conditions with guaranteed levels of safety. To accomplish this, the mission manager (central decision-making module) of the vehicle needs to make informed decisions between various Courses of Action (CoA) based on its' interpretation of the inputs it receives. If an off-nominal event is detected either based on the amalgamation of sensor data or the use of machine learning models, the mission manager may greatly benefit from identification of the input features that most likely contributed to that specific event. Such an understanding is usually not possible to obtain from the classical machine learning models (deep learning) due to the inherent black box like structure. However, this understanding is achieved using eXplainable Artificial Intelligence (XAI) models that provide a human interpretable rationale for the predictions made. This work presents a game theory inspired XAI model for the off-nominal assessment of autonomous flights. The proposed approach based on Shapley values is model agnostic, provides local as well as global explanation and satisfies the four axioms (efficiency, symmetry, dummy, additivity) to achieve fair contribution. The versatility of the approach is first demonstrated on a simulated dataset in which the significance of each input to flight phase prediction is clearly identified. Subsequently, data from simulated flight trajectories are fed into the model which reveal the input features that most likely contributed to a rotor failure event thereby empowering the mission manager to take the appropriate CoA.

autonomy↗

NASA NTRSDate not supplied

Off-Nominal Event Analysis in Autonomous Flights Based on Explainable Artificial Intelligence

A key objective in the Urban Air Mobility program at NASA is to intelligently perform an autonomous flight in a complex urban environment under all weather conditions with guaranteed levels of safety. To accomplish this, the mission manager (central decision-making module) of the vehicle needs to make informed decisions between various Courses of Action (CoA) based on its' interpretation of the inputs it receives. If an off-nominal event is detected either based on the amalgamation of sensor data or the use of machine learning models, the mission manager may greatly benefit from identification of the input features that most likely contributed to that specific event. Such an understanding is usually not possible to obtain from the classical machine learning models (deep learning) due to the inherent black box like structure. However, this understanding is achieved using eXplainable Artificial Intelligence (XAI) models that provide a human interpretable rationale for the predictions made. This work presents a game theory inspired XAI model for the off-nominal assessment of autonomous flights. The proposed approach based on Shapley values is model agnostic, provides local as well as global explanation and satisfies the four axioms (efficiency, symmetry, dummy, additivity) to achieve fair contribution. The versatility of the approach is first demonstrated on a simulated dataset in which the significance of each input to flight phase prediction is clearly identified. Subsequently, data from simulated flight trajectories are fed into the model which reveal the input features that most likely contributed to a rotor failure event thereby empowering the mission manager to take the appropriate CoA.

autonomy↗

NASA NTRSDate not supplied

Biological Research and Space Health Enabled by Machine Learning to Support Deep Space Missions

A key science goal of the NASA “Moon to Mars” campaign is to understand how biology responds to the Lunar, Martian, and deep space environments in order to advance fundamental knowledge, reduce risk, and support safe, productive human space missions. Through the powerful emerging approaches of artificial intelligence (AI) and machine learning (ML), a paradigm shift has begun in biomedical science and engineered astronaut health systems, to enable Earth independence and autonomy of mission operations. Here we present an overview of AI/ML architecture to support deep space mission goals, developed with leaders in the field. First, we focus on the fundamental biological research that supports our understanding of physiological responses to spaceflight, and we describe current efforts to support AI/ML research including data standardization and data engineering through maximally open and FAIR (findable, accessible, interoperable, reusable) databases and the generation of AI-ready datasets for reuse and analysis. We also discuss remote data management frameworks for research data as well as environmental and health data that are generated during deep space missions. We highlight several research projects that leverage data standardization and management for fundamental biological discovery to uncover the complex effects of space travel on living systems. Next, we provide an overview of cutting-edge AI/ML approaches that can be integrated to support remote monitoring and analysis during deep space missions, including generative models and large language models to learn the underlying biomedical patterns and predict outcomes or answer questions during off world medical scenarios. We also describe current AI/ML methods to support this research and monitoring through automated cloud-based labs which enable limited human intervention and closed-loop experimentation in remote settings. These labs could support mission autonomy by analyzing environmental data streams, and would be facilitated through in situ analytics capabilities to avoid sending large raw data files through low bandwidth communications. Finally, in the context of deep space missions with limited communications or access to medical advice from Earth, we describe a solution for integrated, real-time mission biomonitoring across hierarchical levels from continuous environmental monitoring, to wearables and point-of-care devices, to molecular and physiological monitoring. We introduce a precision space health system that will ensure that the future of space health is predictive, preventative, participatory and personalized.

artificial intelligence↗

DOE OSTI2024

Towards FAIR Workflows for Federated Experimental Sciences

A de-centralized, peer-to-peer AI metadata framework is demonstrated which can enable end-to-end metadata & lineage tracking for distributed Machine Learning pipelines spanning edge, High Performance Computing, and cloud environments. With a specific example of end-to-end microscopy algorithm and datasets, the proposed method shows how to enable reproducibility, audit trail, provenance of metadata artifacts. The emerging needs of automation in experimental sciences, ML-centric workflows, and FAIR metadata management across federated compute environments is addressed.

machine learning↗

DOE OSTI2024

Towards Unlocking Insights from Logbooks Using AI

Electronic logbooks contain valuable information about activities and events concerning their associated particle accelerator facilities. However, the highly technical nature of logbook entries can hinder their usability and automation. As natural language processing (NLP) continues advancing, it offers opportunities to address various challenges that logbooks present. This work explores jointly testing a tailored Retrieval Augmented Generation (RAG) model for enhancing the usability of particle accelerator logbooks at institutes like DESY, BESSY, Fermilab, BNL, SLAC, LBNL and CERN. The RAG model uses a corpus built on logbook contributions and aims to unlock insights from these logbooks by leveraging retrieval over facility datasets, including discussion about potential multimodal sources. Our goals are to increase the FAIR-ness (findability, accessibility, interoperability, and reusability) of logbooks by exploiting their information content to streamline everyday use, to enable macro-analysis for root cause analysis, and to facilitate problem-solving automation.

43 PARTICLE ACCELERATORS↗

DOE OSTI2024

Towards Unlocking Insights from Logbooks Using AI

Electronic logbooks contain valuable information about activities and events concerning their associated particle accelerator facilities. However, the highly technical nature of logbook entries can hinder their usability and automation. As natural language processing (NLP) continues advancing, it offers opportunities to address various challenges that logbooks present. This work explores jointly testing a tailored Retrieval Augmented Generation (RAG) model for enhancing the usability of particle accelerator logbooks at institutes like DESY, BESSY, Fermilab, BNL, SLAC, LBNL, and CERN. The RAG model uses a corpus built on logbook contributions and aims to unlock insights from these logbooks by leveraging retrieval over facility datasets, including discussion about potential multimodal sources. Our goals are to increase the FAIR-ness (findability, accessibility, interoperability, and reusability) of logbooks by exploiting their information content to streamline everyday use, enable macro-analysis for root cause analysis, and facilitate problem-solving automation.

43 PARTICLE ACCELERATORS↗

DOE OSTI2021

WA-Omic_LA.1.0 - Quantitative Lipidomics, Metabolomics, and Sequencing (16S/ITS) Publication Data DOI Package

Corresponding Data Publication: "Rapid remodeling of the soil lipidome in response to a drying-rewetting event." This study reveals specific changes in lipids and metabolites that are indicative of stress adaptation, substrate use, and cellular recovery during soil drying and subsequent rewetting. Drought induced nutrient limitation was reflected in the lipidome and polar metabalome, both of which rapidly shifted (within hours) upon rewet. Reduced nutrient access in dry soil caused the replacement of glycerophospholipids with phosphorus-free lipids and impeded resource-expensive osmolyte accumulation. Elevated levels of ceramides and lipids with long chain polyunsaturated fatty acids, in dry soil suggests that lipids play an important role in fungal drought tolerance. Increasing abundance of bacterial glycerophospholipids and triacylglycerols with fatty acids typical of bacteria and polar metabolites suggest metabolic recovery in representative bacteria once the environmental conditions are conducive for growth. These results underscore the importance of the soil lipidome as a robust indicator of microbial community responses, especially at the short time scales of cell-environment reactions. Data package contents reported here are the first version and contain pre- and post-processed data acquisition and subsequent downstream analysis files using various data source instrument method techniques and Mass Spectroscopy (MS) EMSL capabilities. This publication data package DOI is a comprehensive high-throughput multi-omics data lifecycle collection containing processed data method metadata. Support files include additional data download “Read Me” file containing data descriptor information and data source application ontologies (see data dictionary). Reported data download contents are structured for compliance with project data sharing guidelines, community standards initiatives, and sponsor stakeholder policies supporting FAIR data principles. For increased data availability and interoperability, GC-MS/LC-MS mass spectrometry datasets (Thermo .raw ) were deposited at the MassIVE database repository under the related data accession MSV000086931 and can be accessed by using the API. Statistical data processing software, analysis tools, and data workflows are listed below corresponding to the host repository long-term location.

Amplicon sequencing 16S ITS LC-MS/MS lipidomics mu↗

DOE OSTI2021

WA-Omic_LA.1.0 - Quantitative Lipidomics, Metabolomics, and Sequencing (16S/ITS) Publication Data DOI Package

Corresponding Data Publication: "Rapid remodeling of the soil lipidome in response to a drying-rewetting event." This study reveals specific changes in lipids and metabolites that are indicative of stress adaptation, substrate use, and cellular recovery during soil drying and subsequent rewetting. Drought induced nutrient limitation was reflected in the lipidome and polar metabalome, both of which rapidly shifted (within hours) upon rewet. Reduced nutrient access in dry soil caused the replacement of glycerophospholipids with phosphorus-free lipids and impeded resource-expensive osmolyte accumulation. Elevated levels of ceramides and lipids with long chain polyunsaturated fatty acids, in dry soil suggests that lipids play an important role in fungal drought tolerance. Increasing abundance of bacterial glycerophospholipids and triacylglycerols with fatty acids typical of bacteria and polar metabolites suggest metabolic recovery in representative bacteria once the environmental conditions are conducive for growth. These results underscore the importance of the soil lipidome as a robust indicator of microbial community responses, especially at the short time scales of cell-environment reactions. Data package contents reported here are the first version and contain pre- and post-processed data acquisition and subsequent downstream analysis files using various data source instrument method techniques and Mass Spectroscopy (MS) EMSL capabilities. This publication data package DOI is a comprehensive high-throughput multi-omics data lifecycle collection containing processed data method metadata. Support files include additional data download “Read Me” file containing data descriptor information and data source application ontologies (see data dictionary). Reported data download contents are structured for compliance with project data sharing guidelines, community standards initiatives, and sponsor stakeholder policies supporting FAIR data principles. For increased data availability and interoperability, GC-MS/LC-MS mass spectrometry datasets (Thermo .raw ) were deposited at the MassIVE database repository under the related data accession MSV000086931 and can be accessed by using the API. Statistical data processing software, analysis tools, and data workflows are listed below corresponding to the host repository long-term location.

Amplicon sequencing 16S ITS LC-MS/MS lipidomics mu↗

DOE OSTI2021

WA-Omic_LA.1.0 - Quantitative Lipidomics, Metabolomics, and Sequencing (16S/ITS) Publication Data DOI Package

Corresponding Data Publication: "Rapid remodeling of the soil lipidome in response to a drying-rewetting event." This study reveals specific changes in lipids and metabolites that are indicative of stress adaptation, substrate use, and cellular recovery during soil drying and subsequent rewetting. Drought induced nutrient limitation was reflected in the lipidome and polar metabalome, both of which rapidly shifted (within hours) upon rewet. Reduced nutrient access in dry soil caused the replacement of glycerophospholipids with phosphorus-free lipids and impeded resource-expensive osmolyte accumulation. Elevated levels of ceramides and lipids with long chain polyunsaturated fatty acids, in dry soil suggests that lipids play an important role in fungal drought tolerance. Increasing abundance of bacterial glycerophospholipids and triacylglycerols with fatty acids typical of bacteria and polar metabolites suggest metabolic recovery in representative bacteria once the environmental conditions are conducive for growth. These results underscore the importance of the soil lipidome as a robust indicator of microbial community responses, especially at the short time scales of cell-environment reactions. Data package contents reported here are the first version and contain pre- and post-processed data acquisition and subsequent downstream analysis files using various data source instrument method techniques and Mass Spectroscopy (MS) EMSL capabilities. This publication data package DOI is a comprehensive high-throughput multi-omics data lifecycle collection containing processed data method metadata. Support files include additional data download “Read Me” file containing data descriptor information and data source application ontologies (see data dictionary). Reported data download contents are structured for compliance with project data sharing guidelines, community standards initiatives, and sponsor stakeholder policies supporting FAIR data principles. For increased data availability and interoperability, GC-MS/LC-MS mass spectrometry datasets (Thermo .raw ) were deposited at the MassIVE database repository under the related data accession MSV000086931 and can be accessed by using the API. Statistical data processing software, analysis tools, and data workflows are listed below corresponding to the host repository long-term location.

Amplicon sequencing 16S ITS LC-MS/MS lipidomics mu↗

DOE OSTI2020

FaceBase 3: analytical tools and FAIR resources for craniofacial and dental research

ABSTRACT The FaceBase Consortium was established by the National Institute of Dental and Craniofacial Research in 2009 as a ‘big data’ resource for the craniofacial research community. Over the past decade, researchers have deposited hundreds of annotated and curated datasets on both normal and disordered craniofacial development in FaceBase, all freely available to the research community on the FaceBase Hub website. The Hub has developed numerous visualization and analysis tools designed to promote integration of multidisciplinary data while remaining dedicated to the FAIR principles of data management (findability, accessibility, interoperability and reusability) and providing a faceted search infrastructure for locating desired data efficiently. Summaries of the datasets generated by the FaceBase projects from 2014 to 2019 are provided here. FaceBase 3 now welcomes contributions of data on craniofacial and dental development in humans, model organisms and cell lines. Collectively, the FaceBase Consortium, along with other NIH-supported data resources, provide a continuously growing, dynamic and current resource for the scientific community while improving data reproducibility and fulfilling data sharing requirements.

60 APPLIED LIFE SCIENCES↗

NASA NTRS2023

The Historical Greenland Climate Network (GC-Net) Curated and Augmented Level-1 Dataset

The Greenland Climate Network (GC-Net) consists of 31 automatic weather stations (AWSs) at 30 sites across the Greenland Ice Sheet. The first site was initiated in 1990, and the project has operated almost continuously since 1995 under the leadership of the late Konrad Steffen. The GC-Net AWS measured air temperature, relative humidity, wind speed, atmospheric pressure, downward and reflected shortwave irradiance, net radiation, and ice and firn temperatures. The majority of the GC-Net sites were located in the ice sheet accumulation area (17 AWSs), while 11 AWSs were located in the ablation area, and two sites (three AWSs) were located close to the equilibrium line altitude. Additionally, three AWSs of similar design to the GC-Net AWS were installed by Konrad Steffen's team on the Larsen C ice shelf, Antarctica. After more than 3 decades of operation, the GC-Net AWSs are being decommissioned and replaced by new AWSs operated by the Geological Survey of Denmark and Greenland (GEUS). Therefore, making a reassessment of the historical GC-Net AWS data is necessary. We present a full reprocessing of the historical GC-Net AWS dataset with increased attention to the filtering of erroneous measurements, data correction and derivation of additional variables: continuous surface height, instrument heights, surface albedo, turbulent heat fluxes, and 10 m ice and firn temperatures. This new augmented GC-Net level-1 (L1) AWS dataset is now available at https://doi.org/10.22008/FK2/VVXGUT (Steffen et al., 2023) and will continue to be refined. The processing scripts, latest data and a data user forum are available at https://github.com/GEUS-Glaciology-and-Climate/GC-Net-level-1-data-processing (last access: 30 November 2023). In addition to the AWS data, a comprehensive compilation of valuable metadata is provided: maintenance reports, yearly pictures of the stations and the station positions through time. This unique dataset provides more than 320 station years of high-quality atmospheric data and is available following FAIR (findable, accessible, interoperable, reusable) data and code practices.

Greenland Climate Network↗