Search NASA⌕ Search

SEARCH · Search NASA

Results for “Datasets”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,657 records · Page 92
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Characterizing Quantum Classifier Utility in Natural Language Processing Workflows

Quantum Natural Language Processing (QNLP) develops natural language processing (NLP) models for deployment on quantum computers. We explore feature and data prototype selection techniques to address challenges posed by encoding high dimensional features. Our study builds quantum circuit classifiers that includes classical feature pre-processing, quantum embedding and quantum model training. The quantum models are built on 4 or 6 qubits and the quantum neural network (QNN) uses the established bricklayer design. We compare the dependence of model performance (in terms of accuracy and F1 scores) on feature length, embedding gates and parameterized unitary design. We compare the performance of quantum machine learning models to classical convolution neural network model (CNN) on binary and multi-class classification tasks using two datasets of synthetic features and labels. The first is the ECP-CANDLE P3B3 dataset a corpus of synthetically generated cancer pathology reports. The second dataset is extracted from well-known benchmark dataset (MADELON) - features are generated with a combination of informative, repeated and uninformative features. Both datasets are used for binary classification and multi-class classification with 3 classes. We observe robust, accurate performance from all models on the binary classification tasks, but multiclass classification is a challenge for the quantum models-there is a notable decrease in accuracy when using 3 classes. Overall the performance is comparable in terms of recall and accuracy between QNNs and CNNs, even with large datasets. These results provide a point of comparison between quantum and classical models on real-world datasets.

Hamilton, Kathleen↗
DOE OSTI2024

Enhancing Electron Microscopy Image Classification Using Data Augmentation

Manual labeling for machine learning tasks such as image classification is tedious and labor-intensive; as a result, scientific datasets suitable for deep learning applications are scarce and limited. While data augmentation techniques have shown promise for extending image datasets, very little work has been done to understand the impact of combining multiple augmentation methods sequentially or the limits of their effectiveness when combined. Our work addresses this gap by examining how standard and combinatorial data augmentation affects the performance of machine learning models when trained on small datasets for label classification tasks. For our analysis, we generate single, double and quadruple-augmented datasets for a microscopy image classification task using six standard augmentation methods, and compare the resultant improvements observed in binary classification accuracy with three standard image classification models (DenseNet169, MobileNetV2, ResNet101V2). Our experiments show a non-monotonic relationship between the number of simultaneous augmentation methods and classification accuracy, indicating that there is a trade-off between the degree of augmentation and the model performance. These findings suggest that the optimal number of augmentation methods will vary by domain and use case. We also find that the order in which augmentation methods are applied to a limited dataset matters when combining augmentation schemes, with our use case showing performance differences up to 2.6% when the augmentation order is reversed for double-augmented datasets. Our work offers insights to the limits of data augmentation when working on image classification tasks with limited datasets.

Welsman, Jordan A↗
DOE OSTI2020

Deep machine learning-assisted multiphoton microscopy to reduce light exposure and expedite imaging

Two-photon excitation fluorescence (2PEF) allows imaging of tissue up to about one millimeter in thickness. Typically, reducing fluorescence excitation exposure reduces the quality of the image. However, using deep learning super resolution techniques, these low-resolution images can be converted to high-resolution images. This work explores improving human tissue imaging by applying deep learning to maximize image quality while reducing fluorescence excitation exposure. We analyze two methods: a method based on U-Net, and a patch-based regression method. Both methods are evaluated on a skin dataset and an eye dataset. The eye dataset includes 1200 paired high power and low power images of retinal organoids. The skin dataset contains multiple frames of each sample of human skin. High-resolution images were formed by averaging 70 frames for each sample and low-resolution images were formed by averaging the first 7 and 15 frames for each sample. The skin dataset includes 550 images for each of the resolution levels. We track two measures of performance for the two methods: mean squared error (MSE) and structural similarity index measure (SSIM). For the eye dataset, the patches method achieves an average MSE of 27,611 compared to 146,855 for the U-Net method, and an average SSIM of 0.636 compared to 0.607 for the U-Net method. For the skin dataset, the patches method achieves an average MSE of 3.768 compared to 4.032 for the U-Net method, and an average SSIM of 0.824 compared to 0.783 for the U-Net method. Despite better performance on image quality, the patches method is worse than the U-Net method when comparing the speed of prediction, taking 303 seconds to predict one image compared to less than one second for the U-Net method.

62 RADIOLOGY AND NUCLEAR MEDICINE↗
DOE OSTI2021

SMC 2021 Data Challenge: Analyzing Resource Utilization and User Behavior on Titan Supercomputer

Resource utilization statistics of submitted jobs on a supercomputer can help us understand how users from various scientific domains use HPC platforms and better design a job scheduler. We explore to generate insight regarding workload distribution and usage pattern domains from job scheduler trace, GPU failure information, and project-specific information collected from Titan supercomputer. Furthermore, we want to know how the scheduler performance varies over time and how the users' scheduling behavior changes following a system failure. These observations have the potential to provide valuable insight, which is helpful to prepare for system failures. These practices will help us develop and apply novel machine learning algorithms in understanding system behavior, requirement, and better scheduling of HPC systems. There are two datasets, RUR and GPU: RUR dataset is the job scheduler traces collected from the Titan supercomputer from 01/01/2015 to 07/31/2019 (2015.csv - 2019.csv). These were collected using resource Utilization Report (RUR), a Cray-developed resource-usage data collection and reporting system. It contains the usage information of its critical resources (CPU, Memory, GPU, and I/O) of each running job on Titan during that period (https://ieeexplore.ieee.org/abstract/document/8891001). It includes ProjectAreas as additional information, every job is associated with a project ID. TheProjectAreas.csv dataset provides a mapping of the project ID to its domain science. GPU dataset has information regarding GPU failure on Titan. There have been some hardware-related issues in the GPUs in Titan that caused some GPUs to fail, sometimes irrecoverably during some job runs. This dataset provides information regarding these failures during the execution of the submitted jobs. GPUs on Titan are uniquely identified by a serial number (SN), and they are installed in a location. A GPU can be installed in a location, then removed from that location following a failure, and then re-installed in a different location after fixing the problem. If the failure can't be recovered, the GPU might be removed entirely from Titan. There are two prominent types of failures that resulted in the removal of GPUs from Titan: Double Bit Error (DBE) and Out of the Bus (OTB). The dataset (gc_full.csv) has seven attributes, we provided a short description of these attributes in the ReadMe file. To learn more about this dataset, please refer to the git repository https://github.com/olcf/TitanGPULife and the related publication (https://ieeexplore.ieee.org/abstract/document/9355319).

42 ENGINEERING↗
DOE OSTI2023

ORNL_AISD-Ex: Quantum chemical prediction of UV/Vis absorption spectra for over 10 million organic molecules

We performed calculations of electronic excitation energies and associated oscillator strengths based on the time-dependent density-functional tight-binding (TD-DFTB) method [1]. The SMILES (Simplified molecular-input line-entry system) strings of the molecules from the AISD HOMO-LUMO database [2] were converted to a 3D atomistic structure and stored in a PDB file after preliminary geometry optimization using the Merck Molecular Force Field (MMFF94) in RDKit [3,4]. The primary information stored in the PDB file archive consists of Cartesian coordinates for each atom of the molecule in their 3D location in space, along with summary information about the structure, sequence, and experiment. We then performed molecular geometry optimization using the density-functional tight-binding (DFTB) method [5] in the electronic ground state, followed by single-point excited states calculations, as described below. We note that, since RDKit employs a random choice for the generation of molecular conformers, the molecular geometries obtained in this dataset could be different from the ones that were generated when the AISD HOMO-LUMO dataset was generated. The computed excitation energies and associated oscillator strengths can be converted to predict UV/Vis absorption spectra, where excitation energies correspond to absorption peak positions, and oscillator strengths are a good measure of the probability of absorption of visible or UV light in transitions between electronic ground and excited states. The conversion of SMILES strings to 3D Cartesian coordinates of fully DFTB-optimized molecules was successful for 10,502,904 out of 10,502,917 molecules. For these molecules, both geometry optimizations and excited states calculations were successful. The DFTB calculations did not complete for 13 molecules of the original AISD HOMO-LUMO dataset. We still provide information about the geometry of these molecules. The molecules are diverse for chemical compositions (which span 5 non-hydrogen elements: oxygen, carbon, nitrogen, fluorine, sulfur) and molecular size (the smallest molecule contains 5 non-hydrogen atoms, and the largest molecule contains 71 non-hydrogen atoms). The DFTB method [5] is an approximation to density functional theory (DFT), utilizing a minimal basis set in conjunction with a two-center approximation to the electronic Hamiltonian and overlap matrix elements. The DFTB total energy is the sum of an electronic and a repulsive energy contribution, and their calculation requires optimized electronic parameters and diatomic repulsive potential energy functions. All DFTB calculations were performed using the DFTB+ code [6] (version 21.2) and the wrapper for DFTB+ in the Atomic Simulation Environment (ASE) (version 3.22.1) [7], which performed an internal conversion of Cartesian coordinates from PDB to the .gen file format. For the geometry optimizations on the electronic ground state potential energy surface of the molecules, we have chosen the third-order DFTB (DFTB3) method [5c] and employed the matching 3ob set of electronic parameters and repulsive potentials [8]. The empirical γ-damping for hydrogen bond correction, and Grimme's D3 empirical dispersion correction with Becke-Johnson damping (D3(BJ)) [9] dispersion correction was included to improve the description of non-covalent interactions. For excited states single-point energy calculations, we employed the TD-DFTB method in conjunction with the DFTB2 method [5b] and the matching mio [5b,10] and halorg [11] parameter sets. We opted to request the simultaneous calculation of 50 excited states for singlet transition to investigate sufficient number of excited states, based on linear response theory using the Casida equation [Ref: T. A. Niehaus, S. Suhai, F. Della Sala, P Lugli, M. Elstner, G. Seifert, and Th. Frauenheim. Tight-binding approach to time-dependent density-functional response theory. Phys. Rev. B, 63:085108, 2001] and the ARPACK diagonalizer [R. B. Lehoucq, D. C. Sorensen, and C. Yang. Arpack users guide: Solution of large-scale eigenvalue problems by implicitly restarted arnoldi methods, 1997. 46, 51]. The dataset contains 1001 tar.gz files. Tar files are named as “ornl_aisd_ex_1.tar.gz†through “ornl_aisd_ex_1000.tar.gzâ€. Additionally, the 13 failed molecules are in “ornl_aisd_ex_unprocessed.tar.gzâ€. Except for the tar files listed below, each tar file contains 10,500 molecules. Tar files numbered 34, 121, 128, 352, 360, 429, 495, 509, 518, 627, 676, 668, and 862 contain 10,499 molecules each. The last tar file numbered 1000 contains 13,417 molecules. The total size of the uncompressed dataset is over 283 Gigabytes. The code for calculating the electronic excitation energies and statistical analysis of the dataset is provided at the following GitLab repository: https://github.com/ORNL/Analysis-of-Large-Scale-Molecular-Datasets-with-Python Calculating the UV spectrum of a molecule requires performing 3 main operations: 1. Converting the smiles string representation of a molecule into a geometric structure where each atom is assigned XYZ coordinates. The geometric structure is written to the file smiles.pdb. 2. Using smiles.pdb to compute the relaxed geometry of the molecule, which corresponds with the position of the atoms at the position of equilibrium at the ground state. This generates the files band.out, detailed.out, and geo_end.gen. 3. Using geo_end.gen to calculate the UV spectrum of the molecule which is written into the file EXC.DAT. Every molecule in the dataset has its own directory. The files contained in each molecule directory are as follows: 1. geo_end.gen 2. detailed.out 3. band.out 4. EXC.DAT 5. smiles.pdb REFERENCES [1] Niehaus, T. A.; Suhai, S.; Della Salla, F.; Lugli, P.; Elstner, M.; Seifert, G.; Frauenheim, Th. Tight-binding approach to time-dependent density-functional response theory. Phys. Rev. B, 2001, 63, 085108/1-9. [2] Blanchard, A.; Gounley, J.; Metha, K.; Yoo, P.; Irle, S. AISD HOMO-LUMO. DOI: 10.13139/ORNLNCCS/1869409 [3] RDKit: Cheminformatics and Machine Learning Software. 2013, [http://www.rdkit.org] [4] Tosco, P.; Stiefl, N. and Landrum, G. Bringing the MMFF force field to the RDKit: implementation and validation. J Cheminform. 2014, 6, 1–4. [5] a) Porezag, D.; Frauenheim, T.; Kohler, T.; Seifert, G.; Kaschner, Construction of tight-binding-like potentials on the basis of density-functional theory: Application to carbon, R. Phys. Rev. B 1995, 51, 12947-12957; b) Elstner, M.; Porezag, D.; Jungnickel, G.; Elsner, J.; Haugk, M.; Frauenheim, Th.; Suhai, S.; Seifert, G.; Phys. Rev. B 1998, 58, 7260-7268; c) Gaus, M.; Cui, Q.; Elstner, M. DFTB3: Extension of the Self-Consistent-Charge Density-Functional Tight-Binding Method (SCC-DFTB), J. Chem. Theory Comput. 2011, 7, 931-948; d) Cui, Q.; Elstner, M. Density functional tight binding: values of semi-empirical methods in an ab initio era, Phys. Chem. Chem. Phys. 2014, 16, 14368-14377. [6] Hourahine, B. et al. DFTB+, a software package for efficient approximate density functional theory based atomistic simulations, J. Chem. Phys. 2020, 152, 124101/1-19. [7] Larsen, A. H. et al. The atomic simulation environment—a Python library for working with atoms. J. Phys.: Cond. Matter 2017, 29, 273002. [8] Kubillus, M.; Kubar, T.; Gaus, M.; Rezac, J.; Elstner, M. Parameterization of the DFTB3 Method for Br, Ca, Cl, F, I, K, and Na in Organic and Biological Systems, J. Chem. Theory Comput. 2015, 11, 332-342. [9] Brandenburg, J. G.; Grimme, S. Accurate Modeling of Organic Molecular Crystals by Dispersion-Corrected Density Functional Tight Binding (DFTB), J. Phys. Chem. Lett. 2014, 5, 1785−1789. [10] a) Niehaus, T. A.; Elstner, M.; Frauenheim, Th.; Suhai, S. Application of an approximate density-functional method to sulfur containing compounds. J. Mol. Struct.: THEOCHEM 2001, 541, 185-94; b) Elstner, M.; Hobza, P.; Frauenheim, Th.; Suhai, S.; Kaxiras, E. Hydrogen bonding and stacking interactions of nucleic acid base pairs: A density-functional-theory based treatment. J. Chem. Phys. 2001, 114, 5149-55. [11] Kubar, T.; Bodrog, Z.; Gaus, M.; Köhler, C.; Aradi, B.; Frauenheim, Th.; Elstner, M. Parametrization of the SCC-DFTB Method for Halogens. J. Chem. Theory Comput. 2013, 9, 2939-49.

36 MATERIALS SCIENCE↗
DOE OSTI2024

HydraGNN_Predictive_GFM_2024 - Ensemble of predictive graph foundation models for ground state atomistic materials modeling

We provide the ensemble of fifteen pre-trained graph foundation models (GFMs) for atomistic materials modeling applications. Each one of the fifteen GFMs has been trained on five open-source datasets that (once aggregated) amount to over 154 million atomistic structures, which cover over two-thirds of the natural elements of the periodic table and that comprises a broad set of organic and inorganic compounds. This vast set of atomistic structures comprises ground state configurations that are dynamically stable (i.e., equilibrated structures with atomic forces approximately close to zero values) as well as dynamically unstable structures (i.e., non-equilibrium structures with non-negligible non-zero values of atomic forces). The ensemble of datasets aggregated does NOT include excited states. The datasets have been curated to remove atomistic structures with spectral norm of the force tensor above 100 eV/angstrom. Moreover, a linear term of the energy was computed for each dataset using a linear regression model that uses the chemical concentration of each natural element as regressor. The linear term predicted by the linear regression model has been subtracted from each original energy value to perform a re-alignment of the energy values across different electronic structures approximation theories performed to generate the diverse multi-source, multi-fidelity datasets. The folder "ADIOS_files" contains the set of pre-processed datasets in Adaptable I/O System (ADIOS) format (https://www.exascaleproject.org/research-project/adios/) that have been used for the development and training of GFMs in this work. The "ADIOS_files" directory contains 6 sub-directories named as follows: - ANI1x-v3.bp - MPTrj-v3.bp - OC2020-20M-v3.bp - OC2020-v3.bp - OC2022-v3.bp - qm7x-v3.bp Each sub-directory contains the pre-processed datasets converted in Adaptable I/O System (ADIOS) format (https://www.exascaleproject.org/research-project/adios/) that have been used to the development, training, and performance testing of the ensemble go predictive graph foundation models. Each GFM was developed using HydraGNN (https://github.com/ORNL/HydraGNN) as underlying graph neural network (GNN) architecture. The multi-task learning (MTL) capability of HydraGNN was used to simultaneously train the GFMs on labeled values for direct predictions of energy (a total system property of an atomistic structure that measures the chemical stability) and atomic forces (an atomic level property of an atomistic structure that measures the dynamical stability). The hyper parameters of the GFM have been tuned using scalable hyperparameter optimization (HPO) algorithms implemented in the software DeepHyper (https://github.com/deephyper/deephyper). The pre-training of each HPO trial was performed using distributed data parallelism (DDP) to scale the training across 128 compute nodes of the exascale OLCF supercomputer Frontier. Each HPO trial was trained only for 10 epochs and an early stopping was performed to avoid wasting significant computational resources on GNN architectures that were clearly underperforming. For each HPO trial, the 'omnistat' tool developed by (AMD Research - Advanced Micro Device) was used to measure the total energy consumption in kWh. The ensemble of GFMs was obtained by selecting the fifteen best performing HPO trials. Four models have been selected for their clear advantage in accuracy, and these are the GFMs with IDs 229, 156, 147, 260. Additional eleven models have been selected based on judicious balance between accuracy and energy consumption needed for training, and these are the GFMs with IDs 165, 78, 137, 1, 175, 171, 181, 67, 179, 167, 351. Each selected GFM of the ensemble was continued to cumulate a total of at most 30 epochs. In some cases, the total number of epochs actually performed was les than 30 due to two combined factors: (1) the size of the GFM (i.e., the number of model parameters to train) and (2) the total wall-clock time for which the computational resources could be allocated on OLCF-Frontier. The "Ensemble_of_models" directory contains 15 sub-directories named as follows: - gfm_0.229 - gfm_0.156 - gfm_0.147 - gfm_0.260 - gfm_0.165 - gfm_0.78 - gfm_0.137 - gfm_0.1 - gfm_0.175 - gfm_0.171 - gfm_0.181 - gfm_0.67 - gfm_0.179 - gfm_0.167 - gfm_0.351 Each one of these sub-directories refers to one of the fifteen HPO trials that have been selected to continue the pre-training with at most 30 epochs. With each sub-directory associated with a specific HPO trial, the following files can be found: - config.json: file for argument parsing to develop and train an HydraGNN architecture - gfm_0.ID_epoch_N.pk: file with model parameters for HPO ID trial after N epochs of training The ensemble of fifteen GFM architectures was used for (1) ensemble averaging to stabilize the predictions of energy and atomic forces after pre-training for post-processing analysis and (2) ensemble uncertainty quantification (UQ). The code used to develop, pre-train, and load the pre-trained models for post-processing analysis is available on the ORNL-GitHub at the following link: https://github.com/ORNL/HydraGNN/tree/Predictive_GFM_2024

36 MATERIALS SCIENCE↗
DOE OSTI2026

Medium- and Heavy-Duty Truck Duty Cycles

This dataset provides second-by-second duty cycle data for Class 6 and Class 8 diesel trucks in Texas, including key vehicle metrics, engine-related data, and GPS data (excluding GPS latitude and longitude to ensure confidentiality). The data were collected via tablets installed on the trucks and organized into daily datasets, each associated with a unique vehicle ID and date. There are 12 daily datasets for Class 6 diesel trucks (three unique vehicle IDs) and 43 daily datasets for Class 8 diesel trucks (six unique vehicle IDs). The units associated with each column are included in the name. The engine performance data include columns such as engine speed, engine percent torque, and engine fuel rate. Road grade (%/100) was estimated using the GPS altitude and wheel-based vehicle speed, which is used as an input for FASTSim. Cumulative distance was also calculated using the wheel-based vehicle speed. Additional columns include: - Engine Speed (RPM): Removed inaccurate readings and used to calculate angular velocity (radians/second). - Torque (N·m): Calculated using engine percent torque, nominal friction percent torque, and engine reference torque values (those columns were removed from dataset), then normalized to express as torque (%). - Flywheel Power (%): Calculated using the angular velocity and torque (in kW), then normalized as a percentage of the maximum value. - Engine Fuel Rate (%) and Torque (%): Both metrics were normalized by dividing by their respective maximum values within each dataset to express them as percentages. The datasets were analyzed to assess the energy impact of various driving behaviors, simulate energy efficiency, and recommend optimal routes for diesel trucks using NLR’s tool called RouteE. For driver coaching, factors like speed and acceleration limits were considered, and idle periods were reduced (assuming the engine was off during idling) to adjust each drive cycle. These adjusted drive cycles were then simulated in FASTSim to evaluate their effect on fleet energy consumption and estimate potential energy savings. The original cycles are available for download on this page ![image](CoVaR_Image_for_Data_Page_Kenworth_Truck.jpg)

1Hz↗
DOE OSTI2025

Continuous snow depth and temperature measurements from dense network of above-ground distributed temperature profiling systems from 2021-09-23 to 2024-08-23, Seward Peninsula, Alaska

The dataset contains temperature measurements from distributed temperature profiling (DTP) systems (Dafflon et al., 2022; Wielandt et al., 2022; Wang et al., 2024a; Fiolleau et al., 2024) deployed vertically above the ground surface at a large number of locations from 2021 to 2024. The research is designed to improve understanding of the local heterogeneity in snow depth and snow thermal insulation dynamics, as well as their interactions in a discontinuous permafrost region (Wang et al., 2025). The DTP systems were deployed at 96 locations in a watershed along the Nome-Teller road at mile marker 27 (T27) and at 54 locations on a hillslope along the Kougarok road at mile marker 64 (K64) in the Seward Peninsula, Alaska. The probe location information is stored in Probe_locations_T27.csv and Probe_locations_K64.csv. Temperature measurements were recorded at 15-minute intervals using high-precision digital sensors (accuracy: ±0.1°C, resolution: 0.0078°C). The temperature probes, either 1.4 m or 1.6 m long, contain sensors spaced every 5 cm or 10 cm along their length. The temperature data are stored in compressed files following the format: DTP_snow_air_temperature_(site)_(start)_(end).zip, where site is either T27 or K64, and start and end represent the time series period. Within each ZIP file, individual CSV files are named by probe ID and contain temperature records at different heights above the ground surface.This dataset also includes derived snow depth time series over three snow seasons, estimated from temperature measurements. Snow depth was estimated by identifying the consecutive sensor pair that exhibited the largest drop in high-frequency temperature fluctuations (detailed in the methods). These data are stored in: Snow_depths_flags_(site)_(start)_(end).csv, which includes snow depth time series and corresponding quality flags (defined in the methods) from different probes. Additionally, the dataset includes derived metrics and supporting measurements at selected locations over two snow seasons, contributing to the manuscript of Wang et al., 2025. These locations were chosen based on the availability of high-quality snow depth time series during both seasons. The additional data include: (1) Air temperature proxies measured from the top sensors on the pole when they were not buried by snow, stored in Air_temperature_proxies_(site)_(start)_(end).csv (2) Ground interface temperature, recorded at 3 cm above the ground, stored in Ground_interface_temperature_(site)_(start)_(end).csv (3) Site characteristics, including vegetation height, elevation, and the topographic position index (TPI) within a 50 m radius, stored in Selected_probe_locations_gps_vegheight_tpi_elevation_(site).csv. These metrics were derived from 1 m resolution summer LiDAR-based digital elevation models and digital surface models from Singhania et al., 2023, DOI:10.5440/1832016. Metadata files include data descriptions (_dd.csv) for tabular data. All included files are listed and described in xxxx_flmd.csv.This dataset is an updated version of a previous archive (Wang et al., 2024b, DOI: 10.15485/2475020), incorporating multiple seasons and improved snow depth estimation. Please note that due to large amount of information present in this dataset, many specificities associated with the acquisition of snow temperature, air temperature proxy and estimation of snow depth, and the future archiving of additional datasets on the soil temperature, thaw depth and soil characteristics at these locations, the author would welcome being contacted by people planning to use this dataset.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Soil biogeochemical properties and metrics of tree-mycorrhizal dominance for a 25-Ha forest in South Central Indiana, USA.

This data package contains a dataset used in the papers “Seeing the forest for all the trees: Mycorrhizal-associated nutrient economies are modulated by stem density and the synchrony between overstory and understory communities” and “Mycorrhizal associations of tree species influence soil nitrogen dynamics via effects on soil acid–base chemistry”. Four csv files are included along with a dataset. The dataset features chemical soil properties for a single sampling campaign within the 25 Ha Lilly-Dickey Woods Smithsonian Forest Global Earth Observatory (ForestGEO) plot in South Central Indiana, USA (ldw_dat_raw.csv). Also included are separate files focused on pH (pH_data.csv), carbon and nitrogen (CN_data.csv), and nitrification rates (Nitrification_data.csv). These variables are commonly associated with the tree-mycorrhizal dominance of forest stands. In these data subsets, each soil variable was matched to a 10 meter radius neighborhood wherein metrics of tree-mycorrhizal dominance (basal area, stem count, importance value, etc.) were calculated. Models between these soil variables and dominance metrics were used to investigate how different assessments of mycorrhizal associated nutrient economies (MANE) capture these relationships. This research was performed as a part of the Smithsonian ForestGEO project. This data package can be used to explore spatial variability in soil chemistry within a mature hardwood forest, or it can be combined with the included tree data, other fine-scale spatial information, or other tree inventory data for the site to evaluate how soil chemistry varies with tree community composition or edaphic or topographic properties.

Craig, Matthew [ORNL] (ORCID:0000000288907920)↗
DOE OSTI2026

Water Observations of Flow/No-Flow for the East-Taylor Watershed, Colorado (June-July 2025 and 2026)

This dataset provides multi-year, ground-truth visual observations of surface water flow/no-flow conditions within the East-Taylor Watershed, Colorado, collected during June and July of 2025 and 2026. In June and July 2025, on-the-ground visual observations of flow/no-flow were collected as part of the Watershed Function Scientific Focus Area (SFA) and Rocky Mountain Biological Laboratory (RMBL) Colorado Headwaters Ecological Spectroscopy Study (CHESS) campaign (further details are provided within the CHESS Project Description). We obtained 377 water observations of flow/no-flow within the East-Taylor Watershed, Colorado. These ground-truth observations were collected to validate classification maps from remote sensing data and model results within the East-Taylor Watershed. In 2025, flow/no-flow measurements were collected using a field-based app for the CHESS Campaign (Zerion iForm). Within the field app, a water observation form was created to collect coordinates and metadata about the observation. Information collected for the water observation points included information about visually-assessed streamflow presence/absence (standard question obtained from Colorado State University’s StreamTracker project), flow estimate, stream or ponded area width, canopy cover, manganese films, iron seeps, and beaver activity. For 2025 water observations, this dataset contains: (1) a data file with the water observations and coordinates (2025_Water_Observations.csv); (2) a Keyhole Markup Language Zipped (KMZ) with the water observation locations and metadata (2025_Water_Observations_Locations.kmz); (3) photos (.jpg and .jpeg) of the water observation points, organized by location, contained within 2025_Water_Observations_FieldPhotographs.zip file; and (4) water observation protocols and figures (2025_Water_Observation_Protocols.pdf). In June and July 2026, on-the-ground visual observations of flow/no-flow were collected as part of the Watershed Function SFA project. We obtained 365 water observations of flow/no-flow within the East-Taylor Watershed, Colorado. The 2026 observations focused on collecting repeat measurements at the 2025 flow/no-flow observation locations conducted as part of the CHESS campaign. These ground-truth observations were collected to understand differences in flow/no-flow in 2026, given the unprecedented 2026 drought in Colorado. In 2026, flow/no-flow measurements were collected using ArcGIS (Geographic Information System) Survey123. Within the field app, a water observation form was created to collect coordinates and metadata about the observation. Information collected for the water observation points included repeat information from the 2025 water observation effort, including visually-assessed streamflow presence/absence (standard question obtained from Colorado State University’s StreamTracker project), flow estimate, stream or ponded area width, canopy cover, manganese films, iron seeps, beaver activity, and a new metadata component of estimated stream depth (for select locations). For 2026 water observations, this dataset contains: (1) a data file with the water observations and coordinates (2026_Water_Observations.csv); (2) a Keyhole Markup Language Zipped (KMZ) with the water observation locations and metadata (2026_Water_Observations_Locations.kmz); (3) photos (.jpg) of the water observation points, organized by location, contained within 2026_Water_Observations_FieldPhotographs.zip file; and (4) water observation protocols and figures (2026_Water_Observation_Protocols.pdf). For 2025 and 2026 water observations, this dataset contains: (1) a location metadata file (locations.csv); (6) a file-level metadata (flmd.csv) file that lists each file contained in the dataset with associated metadata; and (7) a data dictionary (dd.csv) file that contains column/row headers used throughout the files along with a definition, units, and data type. CHESS Project Description: The Colorado Headwaters Ecological Spectroscopy Study (CHESS) comprised a multi-week airborne remote sensing and field observation campaign in the Upper Gunnison Basin, Colorado, conducted in June and July of 2025. Airborne remote sensing was conducted by the National Ecological Observatory Network Airborne Observation Platform (NEON AOP), concurrent with a field campaign run by the Rocky Mountain Biological Laboratory (RMBL), the Lawrence Berkeley National Laboratory (LBNL) and SLAC National Accelerator Laboratory Watershed Function Science Focus Area (SFA), and NASA-JPL (Jet Propulsion Laboratory) Earth Surface Mineral Dust Source Investigation (EMIT) program. Between June 10 and July 18, 2025, the NEON AOP flight team collected high-resolution aerial imaging spectroscopy and Light Detection and Ranging (LiDAR) data over three domains: the Upper East River (CRBU), Almont Triangle (ALMO), and the Upper Taylor Basin (UPTA). In coordination with the flights, a field campaign acquired ground-truth observations, including observations of vegetation composition, foliar traits, forest demography, and subsurface properties in 18 core sampling areas within the domains. Additional surface water observations were taken at over 380 point locations. All CHESS campaign datasets can be found within the CHESS ESS-DIVE data portal: https://data.ess-dive.lbl.gov/portals/chess. This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231. 2026-09-02: This dataset was updated to include 2026 water observation measurements. The 2025 observation files were also updated to ensure a consistent file naming convention across water observation years.

2018 NEON and 2025 CHESS Campaigns↗
DOE OSTI2026

Monthly Mean In Situ Surface Flux Observations Paired with Satellite-Derived and Reanalysis-Based Flux Data for the Great Lakes Region, 2001–2020

Surface radiative and turbulent heat fluxes over the Great Lakes strongly influence regional hydrological and meteorological processes, and their accurate representation is critical for numerical weather prediction and coupled atmosphere–lake modeling. However, direct flux observations are spatially sparse across the region, so gridded reanalysis and satellite-derived products are often used for climatological analyses and model evaluation despite differences in their flux representations. This dataset provides processed, quality-controlled, monthly mean surface flux observations from the Great Lakes Evaporation Network (GLEN), AmeriFlux, and the National Data Buoy Center, paired with spatiotemporally matched flux estimates from two reanalysis products, the fifth generation European Centre for Medium-Range Weather Forecasts (ECMWF) reanalysis dataset (ERA5) and the Modern Era Reanalysis for Research and Applications, version 2 (MERRA-2), and two satellite-derived products, the Clouds and Earth's Radiant Energy Systems Energy Balanced and Filled (CERES-EBAF) and the Cloud, Albedo and Surface Radiation dataset from AVHRR data - Edition 3 (CLARA-A3). The dataset includes sixteen observational stations with variable temporal coverage within 2001–2020. For each station, a CSV file contains monthly time series of available flux variables, including surface downwelling shortwave radiation (SW), surface downwelling longwave radiation (LW), sensible heat (SH) flux, and latent heat flux (LH), alongside matched gridded product values where available. Columns in the CSV file correspond to different variables sourced from each dataset, with column titles structured as "{dataset}_{variable}". Columns with relevant metadata are also provided in each CSV file, including station latitude and longitude, monthly timestamps, and the name of the sourced observational data. These files are structured for direct use in common analysis tools, including Microsoft Excel, Python pandas, and Python matplotlib. This dataset supports climatological analysis of the Great Lakes regional surface energy budget, evaluation of satellite-derived and reanalysis-based flux products, and development or validation of flux representations in numerical weather prediction and coupled atmosphere–lake models.

Great Lakes↗
DOE OSTI2023

Integration of seismic-pressure-petrophysics inversion of continuous active-seismic monitoring data for monitoring and quantifying CO 2 plume (Final Report)

The overall objective of this project is to develop and validate an integrated package of joint seismic-pressure-petrophysics inversion (jSPPI) of continuous active-source seismic monitoring dataset capable of providing real-time monitoring of CO 2 plume during geologic carbon sequestration (GCS). The three specific developments include: (a) the methodologies for fast seismic full waveform inversion of continuous active source seismic monitoring, (CASSM) datasets for simultaneously estimating velocity and attenuation, and with data assimilation; (b) joint Bayesian petrophysical inversion of seismic models and pressure data for providing and updating CO 2 saturation models; (c) the methods using multiple datasets including (Crainfield and Frio-II borehole) synthetic, laboratory, and field CASSM datasets. The outcomes of jSPPI include (a) a workflow for processing CASSM data, (b) Bayesian inversion algorithms using CASSM data and pressure response data, and (c) integration with data assimilation algorithms for continuously updating site-specific models used for prediction and reservoir management. The validation of joint FWI will be conducted using synthetic models based on the Cranfield and Frio experiments as well as field CASSM datasets collected as part of the Frio-II pilot injection. To quantify and map the mass and distribution of CO 2 (saturation), we will jointly invert velocity and attenuation measurements from the FWI with a Bayesian approach using a rock physics model for attenuation (e.g., White’s attenuation model with two selected patch sizes (White, 1976; Dutta and Seriff, 1979)). The Bayesian inversion will be applied to each time step in the CASSM survey in an updating scheme, which integrates with an ensemble of reservoir simulations at each step. A more complete experimental validation dataset will be collected as part of a mesoscale (2-3 m) gas-CO 2 injection experiment utilizing a higher frequency version of the CASSM system developed for laboratory studies; the integrated inversion will be demonstrated using this dataset which will provide both a dense geometry as well as more precise secondary confirmation measurements (e.g. saturation) typically not available in the field. The resulting real-time map of CO 2 saturation is able to provide a deeper scientific understanding of the complex, time-varying dynamics of subsurface fluid flow migration path as well as the rapid detection of CO 2 leakage hazards.

25 ENERGY STORAGE↗
DOE OSTI2024

Addressing GPU memory limitations for Graph Neural Networks in High-Energy Physics applications

Introduction Reconstructing low-level particle tracks in neutrino physics can address some of the most fundamental questions about the universe. However, processing petabytes of raw data using deep learning techniques poses a challenging problem in the field of High Energy Physics (HEP). In the Exa.TrkX Project, an illustrative HEP application, preprocessed simulation data is fed into a state-of-art Graph Neural Network (GNN) model, accelerated by GPUs. However, limited GPU memory often leads to Out-of-Memory (OOM) exceptions during training, due to the large size of models and datasets. This problem is exacerbated when deploying models on High-Performance Computing (HPC) systems designed for large-scale applications. Methods We observe a high workload imbalance issue during GNN model training caused by the irregular sizes of input graph samples in HEP datasets, contributing to OOM exceptions. We aim to scale GNNs on HPC systems, by prioritizing workload balance in graph inputs while maintaining model accuracy. Our paper introduces diverse balancing strategies aimed at decreasing the maximum GPU memory footprint and avoiding the OOM exception, across various datasets. Results Our experiments showcase memory reduction of up to 32.14% compared to the baseline. We also demonstrate the proposed strategies can avoid OOM in application. Additionally, we create a distributed multi-GPU implementation using these samplers to demonstrate the scalability of these techniques on the HEP dataset. Discussion By assessing the performance of these strategies as data loading samplers across multiple datasets, we can gauge their effectiveness in both single-GPU and distributed environments. Our experiments, conducted on datasets of varying sizes and across multiple GPUs, broaden the applicability of our work to various GNN applications that handle input datasets with irregular graph sizes.

Lee, Claire Songhyun↗
DOE OSTI2021

Open Data and Deep Semantic Segmentation for Automated Extraction of Building Footprints

Advances in machine learning and computer vision, combined with increased access to unstructured data (e.g., images and text), have created an opportunity for automated extraction of building characteristics, cost-effectively, and at scale. These characteristics are relevant to a variety of urban and energy applications, yet are time consuming and costly to acquire with today’s manual methods. Several recent research studies have shown that in comparison to more traditional methods that are based on features engineering approach, an end-to-end learning approach based on deep learning algorithms significantly improved the accuracy of automatic building footprint extraction from remote sensing images. However, these studies used limited benchmark datasets that have been carefully curated and labeled. How the accuracy of these deep learning-based approach holds when using less curated training data has not received enough attention. The aim of this work is to leverage the openly available data to automatically generate a larger training dataset with more variability in term of regions and type of cities, which can be used to build more accurate deep learning models. In contrast to most benchmark datasets, the gathered data have not been manually curated. Thus, the training dataset is not perfectly clean in terms of remote sensing images exactly matching the ground truth building’s foot-print. A workflow that includes data pre-processing, deep learning semantic segmentation modeling, and results post-processing is introduced and applied to a dataset that include remote sensing images from 15 cities and five counties from various region of the USA, which include 8,607,677 buildings. The accuracy of the proposed approach was measured on an out of sample testing dataset corresponding to 364,000 buildings from three USA cities. The results favorably compared to those obtained from Microsoft’s recently released US building footprint dataset.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Development of observation-based global multilayer soil moisture products for 1970 to 2016

Abstract. Soil moisture (SM) datasets are critical to understanding the global water, energy, and biogeochemical cycles and benefit extensive societal applications. However, individual sources of SM data (e.g., in situ and satellite observations, reanalysis, offline land surface model simulations, Earth system model – ESM – simulations) have source-specific limitations and biases related to the spatiotemporal continuity, resolutions, and modeling and retrieval assumptions. Here, we developed seven global, gap-free, long-term (1970–2016), multilayer (0–10, 10–30, 30–50, and 50–100 cm) SM products at monthly 0.5∘ resolution (available at https://doi.org/10.6084/m9.figshare.13661312.v1; Wang and Mao, 2021) by synthesizing a wide range of SM datasets using three statistical methods (unweighted averaging, optimal linear combination, and emergent constraint). The merged products outperformed their source datasets when evaluated with in situ observations (mean bias from −0.044 to 0.033 m3 m−3, root mean square errors from 0.076 to 0.104 m3 m−3, Pearson correlations from 0.35 to 0.67) and multiple gridded datasets that did not enter merging because of insufficient spatial, temporal, or soil layer coverage. Three of the new SM products, which were produced by applying any of the three merging methods to the source datasets excluding the ESMs, had lower bias and root mean square errors and higher correlations than the ESM-dependent merged products. The ESM-independent products also showed a better ability to capture historical large-scale drought events than the ESM-dependent products. The merged products generally showed reasonable temporal homogeneity and physically plausible global sensitivities to observed meteorological factors, except that the ESM-dependent products underestimated the low-frequency temporal variability in SM and overestimated the high-frequency variability for the 50–100 cm depth. Based on these evaluation results, the three ESM-independent products were finally recommended for future applications because of their better performances than the ESM-dependent ones. Despite uncertainties in the raw SM datasets and fusion methods, these hybrid products create added value over existing SM datasets because of the performance improvement and harmonized spatial, temporal, and vertical coverages, and they provide a new foundation for scientific investigation and resource management.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Projections of Hourly Meteorology by Balancing Authority Based on the IM3/HyperFACETS Thermodynamic Global Warming (TGW) Simulations

This dataset contains 40 years (1980-2019) of historical hourly meteorology and 80 years (2020-2099) of projected hourly meteorology for 54 Balancing Authorities (BAs) in the conterminous United States. Details about the scenarios and variables included in this dataset are in the readme.pdf file. This dataset is derived from the IM3/HyperFACETS Thermodynamic Global Warming (TGW) simulations (https://doi.org/10.57931/1885756). More details on the TGW approach can be found at: https://tgw-data.msdlive.org/. If you use this dataset please also cite the raw TGW dataset (Jones, A. D., Rastogi, D., Vahmani, P., Stansfield, A., Reed, K., Thurber, T., Ullrich, P., & Rice, J. S. (2022). IM3/HyperFACETS Thermodynamic Global Warming (TGW) Simulation Datasets (v1.0.0) [Data set]. MSD-LIVE Data Repository. https://doi.org/10.57931/1885756). To go from the TGW data to these BA-level aggregated data we first averaged the raw gridded data by county in the United States. That intermediate data step is also stored in MSD-LIVE (https://doi.org/10.57931/1960548). We then population-weight the county-level hourly data in order to create population-weighted meteorology time series for each BA. Historical populations are from the United States Census Bureau and the evolving future populations are based on Shared Socioeconomic Pathways (SSPs) 3 and 5. The four climate scenarios crossed with the two SSPs yield eight different future projections for each BA: rcp45cooler_ssp3, rcp45cooler_ssp5, rcp45hotter_ssp3, rcp45hotter_ssp5, rcp85cooler_ssp3, rcp85cooler_ssp5, rcp85hotterssp3, rcp85hotterssp5. For the historical period and each of the eight future scenarios the dataset has hourly estimates of the population-weighted average of five meteorological variables: Temperature, specific humidity, shortwave radiation, longwave radiation, and wind speed. All times are in Coordinated Universal Time (UTC). The code to go from the raw TGW data to county-level and then BA-level projections is available at: https://github.com/IMMM-SFA/im3components/tree/main/im3components/wrf_to_tell.

Balancing Authority↗
DOE OSTI2023

Projections of Hourly Meteorology by County Based on the IM3/HyperFACETS Thermodynamic Global Warming (TGW) Simulations

This dataset contains 40 years (1980-2019) of historical hourly meteorology and 80 years (2020-2099) of projected hourly meteorology for each county in the conterminous United States. Details about the scenarios and variables included in this dataset are in the readme.pdf file. This dataset is derived from the IM3/HyperFACETS Thermodynamic Global Warming (TGW) simulations (https://doi.org/10.57931/1885756). More details on the TGW approach can be found at: https://tgw-data.msdlive.org/. If you use this dataset please also cite the raw TGW dataset (Jones, A. D., Rastogi, D., Vahmani, P., Stansfield, A., Reed, K., Thurber, T., Ullrich, P., & Rice, J. S. (2022). IM3/HyperFACETS Thermodynamic Global Warming (TGW) Simulation Datasets (v1.0.0) [Data set]. MSD-LIVE Data Repository. https://doi.org/10.57931/1885756). The four future climate scenarios in the TGW data are: rcp45cooler, rcp45hotter, rcp85cooler, and rcp85hotter. For the historical period and each of the four future scenarios the dataset has hourly estimates of the spatial-average of six meteorological variables for each county: Temperature, specific humidity, shortwave radiation, longwave radiation, and the U (east-west) and V (north-south) components of the wind speed. Times for each file are in the filename and all times are in Coordinated Universal Time (UTC). Counties are identified by their Federal Information Processing Standard (FIPS) code. The mapping between counties and FIPS codes is provided in the state_and_county_fips_codes.csv file. The code to go from the raw TGW data to county-level projections is available at: https://github.com/IMMM-SFA/im3components/tree/main/im3components/wrf_to_tell.

County↗
DOE OSTI2025

Extending Shared Socioeconomic Pathways to Future Water Supply In-frastructure Scenarios: A Case Study of San Antonio, TX

Datasets supporting findings and visualization behind Okoye and McManamay (2025) Extending Shared Socioeconomic Pathways to Future Water Supply Infrastructure Scenarios: A Case Study of San Antonio, TX. Environmental Research Communications, DOI: 10.57931/2563186 These datasets contains the results of a site selection analysis for municipal water supply planning across multiple Shared Socioeconomic Pathways (SSPs 1–5) and hard scenario classification of water systems in San Antonio, TX. It includes data at the resolution of individual surface water supply sources (COMIDs) and integrates a wide range of hydrologic, socioeconomic, infrastructural, and scenario-based planning variables. Please refer to the README file provided in Files for more details. Descriptions of the datasets are provided below. Dataset(s) Descriptions: Dataset_SSP1_SSP4.xlsx - Contains data used for site selection optimization under SSP1 to SSP4. This dataset was generated based on multi-indicator computations (e.g., WAI, WQI, ERI, WTC, WIC), scenario demand projections, and resource and spatial constraints, excluding new reservoir values. Dataset_SSP5.xlsx - Used for site optimization under SSP5. Unlike Dataset_SSP1_SSP4, this dataset includes new reservoir features with updated calculations of WAI, WTC, and WIC to reflect the added infrastructure and supply potential. hard_classification.xlsx - Provides the scenario classification output for each site. Includes both the initial scenario classification based on Euclidean Distance and adjusted classifications based on 30% change reduction BAU.zip - Zipped folder of .shp files showing spatially optimized water supply sites per SSP under the Business-As-Usual (BAU) water demand strategy LowGW.zip - Zipped folder of .shp files showing optimized site selections under the Low Groundwater strategy

geospatial↗