Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmark dataset”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Fundamental limit of jet tagging

Identifying the origin of high-energy hadronic jets (jet tagging) has been a critical benchmark problem for machine learning in particle physics. Jets are ubiquitous at colliders and are complex objects that serve as prototypical examples of collections of particles to be categorized. Over the last decade, machine learning-based classifiers have replaced classical observables as the state of the art in jet tagging. Increasingly complex machine learning models are leading to increasingly more effective tagger performance. Our goal is to address the question of convergence—are we getting close to the fundamental limit on jet tagging or is there still potential for computational, statistical, and physical insights for further improvements? We address this question using state-of-the-art generative models to create a realistic, synthetic dataset with a known jet tagging optimum. Various state-of-the-art taggers are deployed on this dataset, showing that there is a significant gap between their performance and the optimum. Our dataset and software are made public to provide a benchmark task for future developments in jet tagging and other areas of particle physics.

Artificial intelligence↗
DOE OSTI2025

Cross-functional transferability in foundation machine learning interatomic potentials

The rapid development of foundation potentials (FPs) in machine learning interatomic potentials demonstrates the possibility for generalizable learning of the universal potential energy surface. The accuracy of FPs can be further improved by bridging the model from lower-fidelity datasets to high-fidelity ones. In this work, we analyze the challenge of this transfer learning (TL) problem within the CHGNet framework. We show that significant energy scale shifts and poor correlations between GGA and r 2 SCAN hinder cross-functional transferability. By benchmarking different TL approaches on the MP-r 2 SCAN dataset, we demonstrate the importance of elemental energy referencing in the TL of FPs. By comparing the scaling law with and without the pre-training on a low-fidelity dataset, we show that significant data efficiency can still be achieved through TL, even with a target dataset of sub-million structures. We highlight the importance of proper TL and multi-fidelity learning in creating next-generation FPs on high-fidelity data.

Huang, Xu [University of California, Berkeley, CA ↗
DOE OSTI2020

Automated detection of part quality during two-photon lithography via deep learning

Two-photon lithography (TPL) is an additive manufacturing technique for fabricating three-dimensional objects with nanoscale features. A main challenge of TPL is the routine and labor-intensive task of finding suitable light dosage parameters, i.e. writing speed and laser intensity that induce photo-polymerization within a wide variety of candidate photo-curing polymers. Another challenge is the monitoring required during fabrication. In this work, we apply machine learning (ML) models to accelerate the process of identifying optimal light dosage parameters and automate the detection of part quality. We curate TPL videos of different parts fabricated under a range of light dosage parameters using different resins and train spatial-temporal ML models on this data. Our results show that ML models can detect TPL part quality with a 95.1% accuracy in milliseconds. We also evaluate classification failures and identify two operating modes: parameter optimization and part quality detection. Last but not least, we publicly release this labelled dataset so that it may serve as a useful benchmark to the community. Our approach to process optimization and part quality detection addresses important aspects of TPL industrialization, is applicable beyond TPL and should benefit other additive manufacturing techniques with similar barriers to operating at industrial scale.

36 MATERIALS SCIENCE↗
DOE OSTI2022

A three-year dataset supporting research on building energy management and occupancy analytics

Abstract This paper presents the curation of a monitored dataset from an office building constructed in 2015 in Berkeley, California. The dataset includes whole-building and end-use energy consumption, HVAC system operating conditions, indoor and outdoor environmental parameters, as well as occupant counts. The data were collected during a period of three years from more than 300 sensors and meters on two office floors (each 2,325 m 2 ) of the building. A three-step data curation strategy is applied to transform the raw data into research-grade data: (1) cleaning the raw data to detect and adjust the outlier values and fill the data gaps; (2) creating the metadata model of the building systems and data points using the Brick schema; and (3) representing the metadata of the dataset using a semantic JSON schema. This dataset can be used in various applications—building energy benchmarking, load shape analysis, energy prediction, occupancy prediction and analytics, and HVAC controls—to improve the understanding and efficiency of building operations for reducing energy use, energy costs, and carbon emissions.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

A global dataset of terrestrial biological nitrogen fixation

Biological nitrogen fixation (BNF) is the main natural source of new nitrogen inputs in terrestrial ecosystems, supporting terrestrial productivity, carbon uptake, and other Earth system processes. We assembled a comprehensive global dataset of field measurements of BNF in all major N-fixing niches across natural terrestrial biomes derived from the analysis of 376 BNF studies. The dataset comprises 32 variables, including site location, biome type, N-fixing niche, sampling year, quantification method, BNF rate (kg N ha −1 y −1 ), the percentage of nitrogen derived from the atmosphere (%N dfa ), N fixer or N-fixing substrate abundance, BNF rate per unit of N fixer abundance, and species identity. Overall, the dataset combines 1,207 BNF rates for trees, shrubs, herbs, soil, leaf litter, woody litter, dead wood, mosses, lichens, and biocrusts, 152 herb %N dfa values, 1,005 measurements of N fixer or N-fixing substrate abundance, and 762 BNF rates per unit of N fixer abundance for a total of 424 species across 66 countries. This dataset facilitates synthesis, meta-analysis, upscaling, and model benchmarking of BNF fluxes at multiple spatial scales.

Reis Ely, Carla R. [Oregon State Univ., Corvallis,↗
DOE OSTI2021

Fast convolutional neural networks on FPGAs with hls4ml

We introduce an automated tool for deploying ultra low-latency, low-power deep neural networks with convolutional layers on field-programmable gate arrays (FPGAs). By extending the hls4ml library, we demonstrate an inference latency of 5 µs using convolutional architectures, targeting microsecond latency applications like those at the CERN Large Hadron Collider. Considering benchmark models trained on the Street View House Numbers Dataset, we demonstrate various methods for model compression in order to fit the computational constraints of a typical FPGA device used in trigger and data acquisition systems of particle detectors. In particular, we discuss pruning and quantization-aware training, and demonstrate how resource utilization can be significantly reduced with little to no loss in model accuracy. We show that the FPGA critical resource consumption can be reduced by 97% with zero loss in model accuracy, and by 99% when tolerating a 6% accuracy degradation.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

Towards philosophical reasoning with agentic LLMs: Socratic method for scientific assistance

As large language models (LLMs) become central tools in science, improving their reasoning capabilities is critical for meaningful and trustworthy applications. We introduce a Socratic agent for scientific reasoning, implemented through a structured system prompt that guides LLMs via classical principles of inquiry. Unlike typical prompt engineering or retrieval-based methods, our approach leverages definition, analogy, hypothesis elimination, and other Socratic techniques to generate more coherent, critical, and domain-aware responses. We evaluate the agent across diverse scientific domains and benchmark it on the abstraction and reasoning corpus challenge dataset, achieving 97.15% under a fixed prompting protocol and without fine-tuning or external tools. Expert evaluation shows improved reasoning depth, clarity, and adaptability over conventional LLM outputs, suggesting that structured prompting rooted in philosophical reasoning can improve the scientific utility of language models.

LLM reasoning↗
DOE OSTI2025

Structure refinement and anisotropic atomic displacement parameters of 1M Illite: Rietveld and pair distribution function analysis using synchrotron X-ray radiation

Illite, a widespread clay mineral, plays a pivotal role in geological processes, notably as an indicator in diagenetic and hydrothermal alteration environments, and possesses significant industrial relevance in applications including ceramics, construction and catalysis. However, challenges including its nanoscale crystallinity, structural disorder and frequent interstratification with other clay minerals have hindered detailed structural characterization using conventional X-ray diffraction (XRD) techniques. This study employs integrated synchrotron XRD and pair distribution function (PDF) analysis to elucidate the crystal structure of the 1M illite polytype, yielding the first determination of its anisotropic atomic displacement parameters (U aniso ). TheseU aniso parameters provide critical insights into atomic dynamics and static disorder within the structure, enabling a more refined understanding of structure–property relationships. This integrated approach, combining synchrotron XRD, Rietveld refinement and PDF analysis, yields a comprehensive structural characterization, capturing both average crystallographic and local atomic arrangements. Considering illite's widespread geological occurrence and industrial importance, this high-precision structural dataset, especially the determinedU aniso values, provides a crucial benchmark for future modeling and simulation efforts targeting accurate prediction of its physicochemical behavior.

Chemistry↗
DOE OSTI2025

RingX: Scalable Parallel Attention for Long-Context Learning on HPC

The attention mechanism has become foundational for remarkable AI breakthroughs since the introduction of the Transformer, driving the demand for increasingly longer context to power frontier models such as large-scale reasoning language models and high-resolution image/video generators. However, its quadratic computational and memory complexities present substantial challenges. Current state-of-the-art parallel attention methods, such as ring attention, are widely adopted for long-context training but utilize a point-to-point communication strategy that fails to fully exploit the capabilities of modern HPC network architectures. In this work, we propose ringX, a scalable family of parallel attention methods optimized explicitly for HPC systems. By enhancing workload partitioning, refining communication patterns, and improving load balancing, ringX achieves up to 3.4 × speedup compared to conventional ring attention on the Frontier supercomputer. Optimized for both bi-directional and causal attention mechanisms, ringX demonstrates its effectiveness through training benchmarks of a Vision Transformer (ViT) on a climate dataset and a Generative Pre-Trained Transformer (GPT) model, Llama3 8B. Our method attains an end-to-end training speedup of approximately 1.5 × in both scenarios. To our knowledge, the achieved 38% model FLOPs utilization (MFU) for training Llama3 8B with a 1M-token sequence length on 4,096 GPUs represents one of the highest training efficiencies reported for long-context learning on HPC systems. Our code implementation is available at https://github.com/jqyin/ringX-attention.

Yin, Junqi [ORNL] (ORCID:0000000338435520)↗
DOE OSTI2020

Fabricated ecosystem workshop: bridging laboratory to field science. Workshop report.

Lawrence Berkeley National Laboratory (Berkeley Lab) scientists held a workshop at the DOE BER Genomic Sciences PI meeting on the use of Fabricated Ecosystems in Washington D.C. on February 25th, 2020. Ecosystem fabrication is an approach to creating controlled microbial, soil and plant ecologies within a laboratory setting that enable discovery and dissection of environmental variables, activities, and interactions. At Berkeley Lab, we are developing two systems which span spatial and temporal scales — the EcoFAB and the EcoPOD. The participants of the workshop discussed the potential applications and challenges of using fabricated ecosystems as tools to tackle BER-relevant scientific questions. Specifically, they identified (1) research challenges that would benefit from ready access to this infrastructure (2) identified and prioritized technical challenges that currently limit these systems and (3) discussed how to use them to bridge the gap between lab and field research. To ensure that the fabricated ecosystems , in particular the larger-scale EcoPODs, are of use to as much of the BER community as possible, the workshop participants made the following recommendations: 1. Produce publicly available datasets for a set of control variables. Use for benchmarking, and assessing reproducibility between systems. 2. Implement data standards. 3. Develop/leverage nano/micro sensors and in situ root imaging. 4. Encourage the development of interdisciplinary teams to develop EcoPOD experiments. 5. Develop fabricated ecosystems with size and complexity that sits between the EcoFAB and EcoPOD to accelerate use and access.

42 ENGINEERING↗
DOE OSTI2024

An ML-based terrestrial data fusion and augmentation framework to enable advanced understanding of the terrestrial carbon and water interactions

Soil moisture is essential to the terrestrial carbon and water cycles and land–atmosphere interactions. There are various types of soil moisture data, and each type has the distinct spatiotemporal strengths and limitations, depending on the diverse applications and retrieval methodologies of different data types (Li et al., in review; The PNNL-82151 FY23 Report). However, the limitations of different soil moisture data in terms of accuracy and spatiotemporal coverage hinder our ability to further understand the soil moisture dynamics across scales. To have a gap free soil moisture data product with a fine spatiotemporal coverage and vertical profiles, we train extreme gradient boosting (XGBoost) models by using (1) in-situ soil moisture measurements from the International Soil Moisture Network (ISMN), (2) soil moisture from the ECMWF reanalysis (ERA) at the 9 km and sub-daily spatiotemporal resolution, (3) the Daymet meteorological fields, and (4) data products that characterize surface conditions, including soil texture, organic content, topography, vegetation type, and rooting depth. We use the trained XGBoost models that have consistent performance across seven soil layers, i.e., 0–5 cm, 5–10 cm, 10–20 cm, 20–40 cm, 40–60 cm, 60–100 cm, and 100–200 cm, and the gridded model predictors to generate a soil moisture data at the 1 km and daily spatiotemporal resolution for the Continental United States (CONUS) from 2001–2020. This dataset can be broadly used for Earth system model benchmark, monitoring extreme weathers, making informed decisions regarding agriculture, water resource management, climate change mitigation, and ecosystem preservation.

58 GEOSCIENCES↗
DOE OSTI2026

SAM Code Enhancements for Fission Product Tracking of Noble Gases and Metals in MSRs

This report documents fiscal year 2026 enhancements to the System Analysis Module (SAM) for modeling fission product transport in liquid-fueled molten salt reactors (MSRs). The work advances three principal areas: noble gas transport, noble metal deposition, and user interface improvements. The noble gas transport capability integrates drift-flux gas transport, Henry’s law two-film interphase mass transfer with pressure-based nucleation suppression, Knudsen-regime pore diffusion into porous graphite with a conjugate salt-graphite interface constraint, built-in material properties, five Sherwood-number mass transfer correlations including three derived from high-fidelity NekRS simulations, and xenon-135 reactivity feedback through SAM’s point-kinetics model. This work also presents a comprehensive verification test suite, including new analytically verified cases for pressure-dependent onset of interphase gas transfer in a stagnant vertical pipe, a postulated FLiBe-graphite Xe extraction permeator, a gravity riser with a fission-product source, and a descending pipe with gas redissolution driven by hydrostatic pressure. A machine learning framework for bubble rise velocity prediction in molten salt systems is developed and benchmarked on molten-salt and diverse aqueous bubble datasets. The best-performing fine-tuned transfer-learning networks achieve an 82% reduction in RMSE relative to the Clift correlation, and is implemented directly in SAM. The noble metal transport capability is developed, including a liquid-wall deposition model and a gas-surface flotation mechanism that transfers insoluble particles entrained by sparging gas to wetted structures. Verification tests and demonstration cases cover the surface deposition, flotation efflux, and flotation shedding. Finally, a new [SpeciesTransport] input structure replaces positional global vectors with selfcontained, order-independent, named species blocks, simplifies the specification of multiphase species and decay chains, and remains fully compatible with existing SAM input files. Together, these developments improve the physical fidelity, verification basis, and usability of SAM for system-level analyses of fissionproduct behavior in MSRs.

Mui, Travis (ORCID:0000000303736470)↗
DOE OSTI2026

Benchmarking the performance of uncertainty quantification methods for neural network-based interatomic potentials

Machine-learned interatomic potentials (ML-IAPs) continue to gain popularity as accurate, computationally efficient replacements for traditional, physics-based interatomic potentials and expensive ab initio methods. Uncertainty quantification (UQ) of ML-IAPs is a growing area of research as UQ is critical in many applications of IAPs, such as developing curated datasets, active learning-based data augmentation, self-improving models, and estimating the uncertainty of molecular dynamics simulations. In this paper, we construct and benchmark a series of different neural network potentials (NNPs) with varying network architectures to determine the performance of these models with respect to both the mean and uncertainty calibration error. Each NNP method is specifically designed to predict either epistemic or aleatoric uncertainty with particular focus on the differences in behavior between the epistemic and aleatoric uncertainty estimates. We benchmark these methods using multiple datasets common in the ML-IAP literature. The results show that the aleatoric uncertainty from single-shot model architectures is a competitive alternative to ensemble-based epistemic uncertainty predictions in regions of sufficient data-density. However, in regions where the representative data is sparse, aleatoric uncertainty models tend to overpredict and epistemic methods tend to underpredict the actual model error. We conclude that the type of UQ is crucial when discussing performance of probabilistic model results as different methods have different performance characteristics depending on the regime in which they are evaluated. Therefore, the type of UQ method should be carefully evaluated against both the data characteristics and requirements for the intended application.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Assessing Spatial Representativeness of Global Flux Tower Eddy-Covariance Measurements Using Data from FLUXNET2015

Large datasets of carbon dioxide, energy, and water fluxes were measured with the eddy-covariance (EC) technique, such as FLUXNET2015. These datasets are widely used to validate remote-sensing products and benchmark models. One of the major challenges in utilizing EC-flux data is determining the spatial extent to which measurements taken at individual EC towers reflect model-grid or remote sensing pixels. To minimize the potential biases caused by the footprint-to-target area mismatch, it is important to use flux datasets with awareness of the footprint. This study analyze the spatial representativeness of global EC measurements based on the open-source FLUXNET2015 data, using the published flux footprint model (SAFE-f). The calculated annual cumulative footprint climatology (ACFC) was overlaid on land cover and vegetation index maps to create a spatial representativeness dataset of global flux towers. The dataset includes the following components: (1) the ACFC contour (ACFCC) data and areas representing 50%, 60%, 70%, and 80% ACFCC of each site, (2) the proportion of each land cover type weighted by the 80% ACFC (ACFCW), (3) the semivariogram calculated using Normalized Difference Vegetation Index (NDVI) considering the 80% ACFCW, and (4) the sensor location bias (SLB) between the 80% ACFCW and designated areas (e.g. 80% ACFCC and window sizes) proxied by NDVI. Finally, we conducted a comprehensive evaluation of the representativeness of each site from three aspects: (1) the underlying surface cover, (2) the semivariogram, and (3) the SLB between 80% ACFCW and 80% ACFCC, and categorized them into 3 levels. The goal of creating this dataset is to provide data quality guidance for international researchers to effectively utilize the FLUXNET2015 dataset in the future.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Using a surrogate-assisted Bayesian framework to calibrate the runoff-generation scheme in the Energy Exascale Earth System Model (E3SM) v1

Abstract. Runoff is a critical component of the terrestrial water cycle, and Earth system models (ESMs) are essential tools to study its spatiotemporal variability. Runoff schemes in ESMs typically include many parameters so that model calibration is necessary to improve the accuracy of simulated runoff. However, runoff calibration at a global scale is challenging because of the high computational cost and the lack of reliable observational datasets. In this study, we calibrated 11 runoff relevant parameters in the Energy Exascale Earth System Model (E3SM) Land Model (ELM) using a surrogate-assisted Bayesian framework. First, the polynomial chaos expansion machinery with Bayesian compressed sensing is used to construct computationally inexpensive surrogate models for ELM-simulated runoff at 0.5∘ × 0.5∘ for 1991–2010. The error metric between the ELM simulations and the benchmark data is selected to construct the surrogates, which facilitates efficient calibration and avoids the more conventional, but challenging, construction of high-dimensional surrogates for the ELM simulated runoff. Second, the Sobol' index sensitivity analysis is performed using the surrogate models to identify the most sensitive parameters, and our results show that, in most regions, ELM-simulated runoff is strongly sensitive to 3 of the 11 uncertain parameters. Third, a Bayesian method is used to infer the optimal values of the most sensitive parameters using an observation-based global runoff dataset as the benchmark. Our results show that model performance is significantly improved with the inferred parameter values. Although the parametric uncertainty of simulated runoff is reduced after the parameter inference, it remains comparable to the multimodel ensemble uncertainty represented by the global hydrological models in ISMIP2a. Additionally, the annual global runoff trend during the simulation period is not well constrained by the inferred parameter values, suggesting the importance of including parametric uncertainty in future runoff projections.

58 GEOSCIENCES↗
DOE OSTI2020

Data Repository for Power system Open models With Evolving Resources (DR POWER) (Final Scientific/Technical Report)

Pacific Northwest National Laboratory (PNNL) and the National Rural Electric Cooperative Association (NRECA) created a Data Repository for Power system Open models With Evolving Resources (DR POWER) to establish, curate, and evolve open-access power grid models and scenarios, and measurements and observations, collectively referred to as datasets. Existing open-access datasets are inadequate (too small and not representative of the complexity of the modern grid, and scattered all over the world wide web) for the investigation and benchmarking of the advanced optimization methods needed for next-generation grids. Open-access datasets are static and inadequate in other aspects: they do not provide format transformation tools; there is missing information needed for advanced applications; hard to cite; no model evolution history; difficult to update current collections; and little to no community involvement and feedback. Overall, the current landscape is hindering model evolution and benchmarking standards needed for transformational approaches to be developed, tested, and accepted by the power modeling community.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

PYK-SubstitutionOME: an integrated database containing allosteric coupling, ligand affinity and mutational, structural, pathological, bioinformatic and computational information about pyruvate kinase isozymes

Interpreting changes in patient genomes, understanding how viruses evolve and engineering novel protein function all depend on accurately predicting the functional outcomes that arise from amino acid substitutions. To that end, the development of first-generation prediction algorithms was guided by historic experimental datasets. However, these datasets were heavily biased toward substitutions at positions that have not changed much throughout evolution (i.e. conserved). Although newer datasets include substitutions at positions that span a range of evolutionary conservation scores, these data are largely derived from assays that agglomerate multiple aspects of function. To facilitate predictions from the foundational chemical properties of proteins, large substitution databases with biochemical characterizations of function are needed. We report here a database derived from mutational, biochemical, bioinformatic, structural, pathological and computational studies of a highly studied protein family—pyruvate kinase (PYK). A centerpiece of this database is the biochemical characterization—including quantitative evaluation of allosteric regulation—of the changes that accompany substitutions at positions that sample the full conservation range observed in the PYK family. We have used these data to facilitate critical advances in the foundational studies of allosteric regulation and protein evolution and as rigorous benchmarks for testing protein predictions. We trust that the collected dataset will be useful for the broader scientific community in the further development of prediction algorithms.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

A cross-dimensional analysis of data-driven short-term load forecasting methods with large-scale smart meter data

Electricity load forecasting is essential to utility operation and power grid stability. A wide spectrum of data-driven methods, ranging from linear regression models to more recent deep learning models have been adopted to forecast electric load over the years. However, there still lacks a holistic evaluation of the applicability of conventional statistical and machine learning based algorithms with respect to different temporal and spatial scopes, computational requirements, and sensitivity of model-tuning. Enabled by a large-scale electricity load profile dataset of over 40,000 residential customers in a utility region, we conducted a cross-dimensional analysis of data-driven load forecasting methods. Three regression-based and seven deep learning algorithms with different model configurations were evaluated in terms of their overall and peak load prediction accuracy, and training burdens, across spatial aggregation levels ranging from the transformer, feeder, substation, to neighborhood. We found, first, the load forecasting accuracy is constrained by a predictability boundary, influenced by the forecasting horizon and spatial aggregation level. Specifically, RandomForest, XGBoost, TFT, TSMixer, and TiDE models achieved less than 10 % prediction error for up to 96-h ahead forecasting for district, substation, and feeder levels, while other models struggle at long-horizon predictions; Second, for winter and summer peak load dates, most models were able to predict the peak demand timing within ± 1 h, but the prediction percentage error varied by models, with TFT and TiDE models being the top performers; Third, models with similar prediction accuracy can differ in training burden by an order of magnitude. Therefore, choosing model configurations that balance prediction performance and computational resource is an important practical consideration for large-scale deployment of the machine learning based load forecasting. The outcome of this study can guide researchers and practitioners to choose the proper load forecasting algorithms based on their problem scope, required accuracy, and available resources. The predictability boundary can serve as a benchmark for electricity load forecasting problems with new algorithms and datasets.

Li, Han↗