Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmark dataset”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
NASA NTRS2013

Long Hole Film Cooling Dataset for CFD Development : Infrared Thermography and Thermocouple Surveys - Part 1

An experiment investigating flow and heat transfer of long (length to diameter ratio of 18) cylindrical film cooling holes has been completed. In this paper, the thermal field in the flow and on the surface of the film cooled flat plate is presented for nominal freestream turbulence intensities of 1.5 and 8 percent. The holes are inclined at 30deg above the downstream direction, injecting chilled air of density ratio 1.0 onto the surface of a flat plate. The diameter of the hole is 0.75 in. (0.01905 m) with center to center spacing (pitch) of 3 hole diameters. Coolant was injected into the mainstream flow at nominal blowing ratios of 0.5, 1.0, 1.5, and 2.0. The Reynolds number of the freestream was approximately 11,000 based on hole diameter. Thermocouple surveys were used to characterize the thermal field. Infrared thermography was used to determine the adiabatic film effectiveness on the plate. Hotwire anemometry was used to provide flowfield physics and turbulence measurements. The results are compared to existing data in the literature. The aim of this work is to produce a benchmark dataset for Computational Fluid Dynamics (CFD) development to eliminate the effects of hole length to diameter ratio and to improve resolution in the near-hole region. In this report, a Time-Filtered Navier Stokes (TFNS), also known as Partially Resolved Navier Stokes (PRNS), method that was implemented in the Glenn-HT code is used to model coolant-mainstream interaction. This method is a high fidelity unsteady method that aims to represent large scale flow features and mixing more accurately.

Shyam, Vikram↗
DOE OSTI2024

Global 1 km land surface parameters for kilometer-scale Earth system modeling

Earth system models (ESMs) are progressively advancing towards the kilometer scale (“k-scale”). However, the surface parameters for land surface models (LSMs) within ESMs running at the k-scale are typically derived from coarse-resolution and outdated datasets. This study aims to develop a new set of global land surface parameters with a resolution of 1 km for multiple years from 2001 to 2020, utilizing the latest and most accurate available datasets. Specifically, the datasets consist of parameters related to land use and land cover, vegetation, soil, and topography. Differences between the newly developed 1 km land surface parameters and conventional parameters emphasize their potential for higher accuracy due to the incorporation of the most advanced and latest data sources. To demonstrate the capability of these new parameters, we conducted 1 km resolution simulations using the E3SM Land Model version 2 (ELM2) over the contiguous United States. Our results demonstrate that land surface parameters contribute to significant spatial heterogeneity in ELM2 simulations of soil moisture, latent heat, emitted longwave radiation, and absorbed shortwave radiation. On average, about 31 % to 54 % of spatial information is lost by upscaling the 1 km ELM2 simulations to a 12 km resolution. Using eXplainable Machine Learning (XML) methods, the influential factors driving the spatial variability and spatial information loss of ELM2 simulations were identified, highlighting the substantial impact of the spatial variability and information loss of various land surface parameters, as well as the mean climate conditions. The comparison against four benchmark datasets indicates that ELM generally performs well in simulating soil moisture and surface energy fluxes. The new land surface parameters are tailored to meet the emerging needs of k-scale LSM and ESM modeling with significant implications for advancing our understanding of water, carbon, and energy cycles under global change. The 1 km land surface parameters are publicly available at https://doi.org/10.5281/zenodo.10815170 (Li et al., 2024).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Model Assumptions and Data Characteristics: Impacts on Domain Adaptation in Building Segmentation

Studies on domain adaptation (DA) for remote sensing (RS) imagery analysis lack consistency in selection and description of evaluation scenarios. Without properly characterizing datasets, model assumptions, and evaluation scenarios, it is difficult to objectively compare DA methods and reach conclusions about their suitability across different applications. With this motivation, this work seeks to empirically assess to which extent the interaction between data characteristics and model assumptions influences the effectiveness of DA methods. Using the widely explored task of building footprint segmentation as a case study, we perform a large-scale study across over 200 DA scenarios that include variations across view angles, areas observed, and sensors used for data acquisition. Rather than adopting different model architectures or optimization criteria, we contrast the performances of two DA methods based on adversarial learning that differ only in their assumptions about source and target domains. Informed by metadata and data characteristics unveiled using traditional computer vision (CV) techniques as well as pretrained deep models, we provide a detailed meta-analysis of experiments highlighting the importance of accurately considering data assumptions for DA in RS segmentation tasks. As demonstrated by a “cherry-picking” exercise, different claims regarding which model is best could be made by selecting different subsets of evaluation scenarios. While well-calibrated assumptions can be beneficial, mismatching assumptions can lead to negative biases in DA applications. Furthermore, this study intends to motivate the community toward more consistent evaluation protocols while providing recommendations and insights toward creating novel benchmark datasets, documenting data characteristics, application-specific knowledge, and model assumptions.

42 ENGINEERING↗
DOE OSTI2025

Adiabatic quantum support vector machines

Adiabatic quantum computers can solve difficult optimization problems (e.g., the quadratic unconstrained binary optimization problem), and they seem well suited to train machine learning models. In this paper, we describe an adiabatic quantum approach for training support vector machines. We show that the time complexity of our quantum approach is an order of magnitude better than the classical approach. Next, we compare the test accuracy of our quantum approach against a classical approach that uses the Scikit-learn library in Python across five benchmark datasets (Iris, Wisconsin Breast Cancer (WBC), Wine, Digits, and Lambeq). We show that our quantum approach obtains accuracies on par with the classical approach. Finally, we perform a scalability study in which we compute the total training times of the quantum approach and the classical approach with an increasing number of features and an increasing number of data points in the training dataset. In conclusion, our scalability results show that the quantum approach obtains a 3.5–4.5x speedup over the classical approach on datasets with many (millions of) features.

Computational Complexity↗
DOE OSTI2021

A scalable algorithm for the optimization of neural network architectures

In this work, we propose a new scalable method to optimize the architecture of an artificial neural network. The proposed algorithm, called Greedy Search for Neural Network Architecture, aims to determine a neural network with minimal number of layers that is at least as performant as neural networks of the same structure identified by other hyperparameter search algorithms in terms of accuracy and computational cost. Numerical results performed on benchmark datasets show that, for these datasets, our method outperforms state-of-the-art hyperparameter optimization algorithms in terms of attainable predictive performance by the selected neural network architecture, and time-to-solution for the hyperparameter optimization to complete.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Sensitivity evaluation of the Kudryavtsev permafrost model

Modeling is an important way to assess current and future permafrost spatial distribution and dynamics, especially in data poor areas like the Arctic region. Here, we evaluate a physics-based analytical model, Kudryavtsev's active layer model, which is widely used because it has relatively few data requirements. This model was recently incorporated into a component modeling toolbox, allowing for coupled modeling of permafrost and geomorphic processes over geological timescales. However, systematic quantitative assessment of the influence of its controlling parameters on permafrost temperature and active layer thickness predictions has not been undertaken before. In this work, we investigate the sensitivity of the Kudryavtsev's active layer model by Monte Carlo simulations to generate probability distributions for input parameters and compare predictions with a comprehensive benchmark dataset of in-situ permafrost observations over entire Alaska. Predicted permafrost surface temperature is highly dependent on mean annual air temperature (r = 0.78 on average), annual temperature amplitude (–0.41), and winter-averaged snow thickness (0.30). Uncertainty of predicted permafrost temperature is relatively small (RMSE = 1 °C), when air temperature and snow depth are well constrained. Similarly, RMSE between simulated and observed ALT at stations is ~0.08 m. However, under given air temperature and snow conditions, soil water content bias can significantly affect modeled active layer thickness (RMSE = 0.1 m or 40% of the observed active layer thickness). If soil water content has a large bias, improvements in other parameters may not significantly improve the active layer predictions of the Kudryavtsev's model.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Score-based denoising for atomic structure identification

We propose an effective method for removing thermal vibrations that complicate the task of analyzing complex dynamics in atomistic simulation of condensed matter. Our method iteratively subtracts thermal noises or perturbations in atomic positions using a denoising score function trained on synthetically noised but otherwise perfect crystal lattices. The resulting denoised structures clearly reveal underlying crystal order while retaining disorder associated with crystal defects. Purely geometric, agnostic to interatomic potentials, and trained without inputs from explicit simulations, our denoiser can be applied to simulation data generated from vastly different interatomic interactions. The denoiser is shown to improve existing classification methods, such as common neighbor analysis and polyhedral template matching, reaching perfect classification accuracy on a recent benchmark dataset of thermally perturbed structures up to the melting point. Demonstrated here in a wide variety of atomistic simulation contexts, the denoiser is general, robust, and readily extendable to delineate order from disorder in structurally and chemically complex materials.

36 MATERIALS SCIENCE↗
DOE OSTI2023

AI-accelerated protein-ligand docking for SARS-CoV-2 is 100-fold faster with no significant change in detection

Protein-ligand docking is a computational method for identifying drug leads. The method is capable of narrowing a vast library of compounds down to a tractable size for downstream simulation or experimental testing and is widely used in drug discovery. While there has been progress in accelerating scoring of compounds with artificial intelligence, few works have bridged these successes back to the virtual screening community in terms of utility and forward-looking development. We demonstrate the power of high-speed ML models by scoring 1 billion molecules in under a day (50 k predictions per GPU seconds). We showcase a workflow for docking utilizing surrogate AI-based models as a pre-filter to a standard docking workflow. Our workflow is ten times faster at screening a library of compounds than the standard technique, with an error rate less than 0.01% of detecting the underlying best scoring 0.1% of compounds. Our analysis of the speedup explains that another order of magnitude speedup must come from model accuracy rather than computing speed. In order to drive another order of magnitude of acceleration, we share a benchmark dataset consisting of 200 million 3D complex structures and 2D structure scores across a consistent set of 13 million “in-stock” molecules over 15 receptors, or binding sites, across the SARS-CoV-2 proteome. We believe this is strong evidence for the community to begin focusing on improving the accuracy of surrogate models to improve the ability to screen massive compound libraries 100 × or even 1000 × faster than current techniques and reduce missing top hits. The technique outlined aims to be a fast drop-in replacement for docking for screening billion-scale molecular libraries.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Enabling high-throughput enzyme discovery and engineering with a low-cost, robot-assisted pipeline

Abstract As genomic databases expand and artificial intelligence tools advance, there is a growing demand for efficient characterization of large numbers of proteins. To this end, here we describe a generalizable pipeline for high-throughput protein purification using small-scale expression in E. coli and an affordable liquid-handling robot. This low-cost platform enables the purification of 96 proteins in parallel with minimal waste and is scalable for processing hundreds of proteins weekly per user. We demonstrate the performance of this method with the expression and purification of the leading poly(ethylene terephthalate) hydrolases reported in the literature. Replicate experiments demonstrated reproducibility and enzyme purity and yields (up to 400 µg) sufficient for comprehensive analyses of both thermostability and activity, generating a standardized benchmark dataset for comparing these plastic-degrading enzymes. The cost-effectiveness and ease of implementation of this platform render it broadly applicable to diverse protein characterization challenges in the biological sciences.

36 MATERIALS SCIENCE↗
DOE OSTI2025

The design space of E(3)-equivariant atom-centred interatomic potentials

Abstract Molecular dynamics simulation is an important tool in computational materials science and chemistry, and in the past decade it has been revolutionized by machine learning. This rapid progress in machine learning interatomic potentials has produced a number of new architectures in just the past few years. Particularly notable among these are the atomic cluster expansion, which unified many of the earlier ideas around atom-density-based descriptors, and Neural Equivariant Interatomic Potentials (NequIP), a message-passing neural network with equivariant features that exhibited state-of-the-art accuracy at the time. Here we construct a mathematical framework that unifies these models: atomic cluster expansion is extended and recast as one layer of a multi-layer architecture, while the linearized version of NequIP is understood as a particular sparsification of a much larger polynomial model. Our framework also provides a practical tool for systematically probing different choices in this unified design space. An ablation study of NequIP, via a set of experiments looking at in- and out-of-domain accuracy and smooth extrapolation very far from the training data, sheds some light on which design choices are critical to achieving high accuracy. A much-simplified version of NequIP, which we call BOTnet (for body-ordered tensor network), has an interpretable architecture and maintains its accuracy on benchmark datasets.

Computer Science↗
DOE OSTI2025

Assessing the potential of deep learning for protein–ligand docking

The effects of ligand binding on protein structures and their in vivo functions carry numerous implications for modern biomedical research and biotechnology development efforts such as drug discovery. Although several deep learning (DL) methods and benchmarks designed for protein–ligand docking have recently been introduced, so far no previous works have systematically studied the behaviour of the latest docking and structure prediction methods within the broadly applicable context of: (1) using predicted (apo) protein structures for docking (for example, for applicability to new proteins); (2) binding multiple (cofactor) ligands concurrently to a given target protein (for example, for enzyme design); and (3) having no previous knowledge of binding pockets (for example, for generalization to unknown pockets). To enable a deeper understanding of the real-world utility of docking methods, we introduce PoseBench, a comprehensive benchmark for broadly applicable protein–ligand docking. PoseBench enables researchers to rigorously and systematically evaluate DL methods for apo-to-holo protein–ligand docking and protein–ligand structure prediction using both primary ligand and multiligand benchmark datasets, the latter of which we introduce to the DL community. Empirically, using PoseBench, we find that: (1) DL cofolding methods generally outperform comparable conventional and DL docking baseline algorithms, but popular methods such as AlphaFold 3 are still challenged by prediction targets with new protein–ligand binding poses; (2) certain DL cofolding methods are highly sensitive to their input multiple sequence alignments, whereas others are not; and (3) DL methods struggle to strike a balance between structural accuracy and chemical specificity when predicting new or multiligand protein targets.

Morehead, Alex [Lawrence Berkeley National Laborat↗
DOE OSTI2024

Uncertainty quantification for molecular property predictions with graph neural architecture search

Graph Neural Networks (GNNs) have emerged as a prominent class of data-driven methods for molecular property prediction. However, a key limitation of typical GNN models is their inability to quantify uncertainties in the predictions. This capability is crucial for ensuring the trustworthy use and deployment of models in downstream tasks. To that end, we introduce AutoGNNUQ, an automated uncertainty quantification (UQ) approach for molecular property prediction. AutoGNNUQ leverages architecture search to generate an ensemble of high-performing GNNs, enabling the estimation of predictive uncertainties. Our approach employs variance decomposition to separate data (aleatoric) and model (epistemic) uncertainties, providing valuable insights for reducing them. In our computational experiments, we demonstrate that AutoGNNUQ outperforms existing UQ methods in terms of both prediction accuracy and UQ performance on multiple benchmark datasets, and generalizes well to out-of-distribution datasets. Additionally, we utilize t-SNE visualization to explore correlations between molecular features and uncertainty, offering insight for dataset improvement. AutoGNNUQ has broad applicability in domains such as drug discovery and materials science, where accurate uncertainty quantification is crucial for decision-making.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

Too big, too small, or just right? A benchmark assessment of density functional theory for predicting the spatial extent of the electron density of small chemical systems

Multipole moments are the first-order responses of the energy to spatial derivatives of the electric field strength. The quality of density functional theory prediction of molecular multipole moments thus characterizes errors in modeling the electron density itself, as well as the performance in describing molecules interacting with external electric fields. However, only the lowest non-zero moment is translationally invariant, making the higher-order moments origin-dependent. Therefore, instead of using the 3 × 3 quadrupole moment matrix, we utilize the translationally invariant 3 × 3 matrix of second cumulants (or spatial variances) of the electron density as the quantity of interest (denoted by K). The principal components of K are the square of the spatial extent of the electron density along each axis. A benchmark dataset of the principal components of K for 100 small molecules at the coupled cluster singles and doubles with perturbative triples at the complete basis set limit is developed, resulting in 213 independent K components. The performance of 47 popular and recent density functionals is assessed against this Var213 dataset. Several functionals, especially double hybrids, and also SCAN and SCAN0 predict reliable second cumulants, although some modern, empirically parameterized functionals yield more disappointing performance. The H, Li, and Be atoms, in particular, are challenging for nearly all methods, indicating that future functional development could benefit from the inclusion of their density information in training or testing protocols.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

Study of self-interaction-errors in barrier heights using locally scaled and Perdew-Zunger self-interaction methods

We study the effect of self-interaction errors on the barrier heights of chemical reactions. For this purpose we use the well-known Perdew-Zunger [J. P. Perdew and A. Zunger, Phys. Rev. B,23, 5048 (1981)] self-interaction-correction(PZSIC), as well two variations of the recently developed, locally scaled self-interaction correction (LSIC) [Zope et al.,J. Chem. Phys.151, 214108 (2019)] to study the barrier heights of the BH76 benchmark dataset. Our results show that both PZSIC and especially the LSIC methods improve the barrier heights relative to the local density approximation(LDA). The version of LSIC that uses the iso-orbital indicator z as a scaling factor gives a more consistent improvement than an alternative version that uses an orbital-dependent factor based on the ratio of orbital densities to the total electron density. We show that LDA energies evaluated using the self-consistent and self-interaction-free PZSIC densities can be used to assess density-driven errors. Furthermore, the LDA reaction barrier errors for the BH76 set are found to contain significant density-driven errors for all types of reactions contained in the set, but the corrections due to adding SIC to the functional are much larger than those stemming from the density for the hydrogen transfer reactions and of roughy equal size for the non-hydrogen transfer reactions.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

Molecular-orbital-based machine learning for open-shell and multi-reference systems with kernel addition Gaussian process regression

We introduce a novel machine learning strategy, kernel addition Gaussian process regression (KA-GPR), in molecular-orbital-based machine learning (MOB-ML) to learn the total correlation energies of general electronic structure theories for closed- and open-shell systems by introducing a machine learning strategy. The learning efficiency of MOB-ML(KA-GPR) is the same as the original MOB-ML method for the smallest criegee molecule, which is a closed-shell molecule with multi-reference characters. In addition, the prediction accuracies of different small free radicals could reach the chemical accuracy of 1 kcal/mol by training on one example structure. Accurate potential energy surfaces for the H10 chain (closed-shell) and water OH bond dissociation (open-shell) could also be generated by MOB-ML(KA-GPR). To explore the breadth of chemical systems that KA-GPR can describe, we further apply MOB-ML to accurately predict the large benchmark datasets for closed- (QM9, QM7b-T, and GDB-13-T) and open-shell (QMSpin) molecules.

Chemistry↗
DOE OSTI2025

A comparative analysis of YOLOv8 and U-Net image segmentation approaches for transmission electron micrographs of polycrystalline thin films

Metallic thin films offer a platform to experimentally study the dynamics of microstructural evolution, but the required transmission electron microscopy (TEM)-based imaging generates complex images that are challenging to segment and quantify. This work provides a comparative analysis of a new YOLOv8 model and an established U-Net model for bright-field TEM images of polycrystals, employing a framework leveraging physical observables to evaluate performance against two hand-traced benchmark datasets. This methodology obviates the comparison of large, diversely structured, and manually labeled datasets that are required to assess performance on a per-image/per-pixel basis. It is found that the YOLOv8 model, adapted for real-time instance segmentation, has up to 43× faster inferencing (NVIDIA GeForce RTX 4090) compared to U-Net and reconstructs hand-traced grain size distributions (GSDs) with excellent fidelity, finding mean diameter within 3% for grains near an optimal magnification; for grains that deviate from the optimal pixel-diameter, the size of small- (large)-diameter grains is systematically over- (under)-estimated. This is partially mitigated by including scale-aware augmentations during training. Moreover, when the bias is corrected post-inference by a rigid shift in distribution, the YOLOv8 model reproduces ground truth GSDs with exceptional fidelity, with statistical tests indicating <5% probability that the distributions are distinct. Based on ground truth data, calibration curves pertaining to this shift can be constructed for a given model. This issue is not present in the U-Net model’s results, indicating that for quantitative measurements where the true size of objects is of interest, special procedures must be implemented for YOLO-based models.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Energy metric prediction for double insertion mutants via the RoseNet deep learning framework

Studying the structural and functional implications of protein mutations is an important task in computational biology and bioinformatics. We leverage our previously proposed RoseNet neural network architecture to predict energy metrics of proteins with double amino acid insertions or deletions (InDels). We train models on previously generated benchmark datasets containing the exhaustive double InDel mutations for three proteins, as well as an additional three proteins for which ∼145k random mutants, each with two InDels, have been generated. We expand on our previous work by evaluating three additional proteins and analyzing domain features that impact the prediction capabilities of RoseNet. These features include InDels into secondary structures and the solvent accessible surface area (SASA) scores of the residues. We uncover further evidence to support that RoseNet has a higher proficiency of generalizing to unseen residue combinations than unseen insertion positions. We also observe that RoseNet produces higher-quality predictions when inserting into a β-sheet over an α-helix. Additionally, when the insertions fall in an area of high SASA, RoseNet often displays better performance than inserting into areas of low SASA.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Isolating Unisolated Upsilons with Anomaly Detection in CMS Open Data

We present the first study of anti-isolated Upsilon decays to two muons (ϒ→𝜇⁺⁢𝜇⁻) in proton-proton collisions at the Large Hadron Collider. Using a machine learning (ML)-based anomaly detection strategy, we “rediscover” the ϒ in 13 TeV CMS Open Data from 2016, despite overwhelming anti-isolated backgrounds. We elevate the signal significance to 6.4⁢𝜎 using these methods, starting from 1.6⁢𝜎 using the dimuon mass spectrum alone. Moreover, we demonstrate improved sensitivity from using an ML-based estimate of the multifeature likelihood compared to traditional “cut-and-count” methods. This is the first ever detection of anti-isolated Upsilons, which can be useful in the study of heavy-flavor fragmentation in quantum chromodynamics. Our Letter demonstrates that it is possible and practical to find real signals in experimental collider data using ML-based anomaly detection, and we distill a readily accessible benchmark dataset from the CMS Open Data to facilitate future anomaly detection developments.

machine learning↗