Search NASA⌕ Search

SEARCH · Search NASA

Results for “BENCHMARK”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Benchmarking Coordination Number Prediction Algorithms on Inorganic Crystal Structures

Coordination numbers and geometries form a theoretical framework for understanding and predicting materials properties. Algorithms to determine coordination numbers automatically are increasingly used for machine learning and automatic structural analysis. In this work, we introduce MaterialsCoord, a benchmark suite containing 56 experimentally-derived crystal structures (spanning elements, binaries, and ternary compounds) and their corresponding coordination environments as described in the research literature. We also describe CrystalNN, a novel algorithm for determining near neighbors. We compare CrystalNN against 7 existing near-neighbor algorithms on the MaterialsCoord benchmark, finding CrystalNN to perform similarly to several well-established algorithms. For each algorithm, we also assess computational demand and sensitivity towards small perturbations that mimic thermal motion. Finally, we investigate the similarity between bonding algorithms when applied to the Materials Project database. Finally, we expect that this work will aid the development of coordination prediction algorithms as well as improve structural descriptors for machine learning and other applications.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

A Theoretical Benchmark of the Geometric and Optical Properties for 3d Transition Metal Nanoclusters via Density Functional Theory

Understanding structure–property relationships in atomically precise metal nanoclusters is vital in finding selective and tunable catalysts. Here, in this study, density functional theory (DFT) was used to benchmark seven exchange correlation functionals at different basis sets for 17 atomically precise nanoclusters against experimentally determined geometries, band gaps, and optical gaps. The set contains both monometallic and bimetallic clusters that possess at least two types of 3d transition metals (specifically, Cu, Ni, Fe, or Co). The benchmark highlights that PBE0 is a good functional to use regardless of the basis set, and Minnesota functionals do well with respect to specific metals. Further, while long-range corrected functionals overestimate band and optical gaps, they model absorption features better than the other considered functionals. The study additionally looks at the photoinduced hydrogen evolution reaction (HER) and the CO 2 reduction mechanism on nanoclusters reported from the literature.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2026

Concordant Mode Approach (CMA): Vibrational Analysis of New and Upgraded Intermolecular Benchmarks for Noncovalent Bonding

The Concordant Mode Approach (CMA) is a novel method that offers tremendous potential for increasing the system size and the level of theory attainable in quantum chemical computations of molecular vibrational frequencies. To investigate the extension of CMA to intermolecular vibrations, computations with coupled cluster singles and doubles with perturbative triples theory [CCSD(T)] using two augmented correlation-consistent polarized-valence triple-ζ basis sets (aug-cc-pVTZ or h-aug-ccpVTZ) were performed on 17 prototypical loosely bound complexes of hydrogen-bonded, dispersion, and mixed character. These Level A results provide new and upgraded benchmarks for noncovalent bonding and a severe test for CMA vibrational analyses. The Level A target frequencies were recovered remarkably well using second-order Møller−Plesset perturbation theory (MP2) with h-aug-cc-pVTZ for generating the underlying (Level B) normal modes of the CMA scheme. Employing this Level B within the lowest-rung CMA-0A method reproduces the 435 benchmark frequencies with a mean absolute error (MAE) of 0.23 cm −1 and a corresponding standard deviation (σ) of 0.84 cm −1 ; strikingly, the corresponding subset of 106 interfragment frequencies exhibits MAE = 0.34 cm −1 and σ = 0.90 cm −1 . Subsequent application of the higher-rung CMA-2A scheme eliminates all outliers and reduces the overall MAE to a minuscule 0.08 cm −1 with the inclusion of only 3.0% of the off-diagonal couplings not accounted for by CMA-0A. Accordingly, the highly efficient CMA methodology proves to be robust even for vibrations on flat potential energy surfaces.

Aromatic compounds↗
DOE OSTI2025

Diffusion Quantum Monte Carlo Benchmarking of Magnetic Moments in MnBi 2 Te 4

The intrinsically antiferromagnetic topological insulator, MnBi 2 Te 4 (MBT), has garnered significant attention recently due to its potential to host numerous exotic topological quantum states. Unfortunately, their consistent realization has been hindered by intrinsic antisite defects among the Mn and Bi sublattices. In this work, we establish Mn magnetization of pristine MBT through high level diffusion Monte Carlo calculations, which can serve as a precise starting point for various models to estimate antisite defect concentrations in actual MBT samples. The benchmark quality of DMC calculations is further identified from out model estimating antisite defect concentrations, which combines the benchmarked Mn magnetization with data from magnetic susceptibility and intermediate field magnetization measurements. This reproduces well Bi Mn and Mn Bi concentrations measured in the experiments. Here, we anticipate these theoretically based magnetic purity measures may be used as minimization targets in cycles of refinement to synthesize MBT with low antisite defect concentrations and more reproducible topological properties.

Defects↗
DOE OSTI2021

Guest-Host Interactions in Clathrate Hydrates: Benchmark MP2 and CCSD(T)/CBS Binding Energies of CH4, CO2 and H2S in (H2O)20 Cages

We present benchmark binding energies of naturally occurring gas molecules CH4, CO2, and H2S in the small cage, namely the pentagonal dodecahedron (512) (H2O)20, which is one of the constituent cages of the 3 major lattices (structures I, II and H) of clathrate hydrates. These weak interactions require higher levels of electron correlation and converge slowly with increasing basis set to the Complete Basis Set (CBS) limit, necessitating the use of large basis sets up to the augcc- pV5Z and subsequent correction for Basis Set Superposition Error (BSSE). For the host hollow (H2O)20 cages we have identified a most stable isomer with binding energy of -200.8 ± 2.1 kcal/mol at the CCSD(T)/CBS limit (-199.2 ± 0.5 kcal/mol at the MP2/CBS limit). Additionally, we report converged second order Moller-Plesset (MP2) CBS binding energies for the encapsulation of guests in the (H2O)20 cage of -4.3 ± 0.1 for CH4@(H2O)20, -6.6 ± 0.1 for CO2@(H2O)20 and -8.5 ± 0.1 kcal/mol for H2S@(H2O)20, respectively. For CH4@(H2O)20, exhibiting the weakest encapsulation affinity among the three, we report CCSD(T)/aug-cc-pVTZ binding energies and, based on them, a CCSD(T)/CBS estimate of -4.75 ± 0.1 kcal/mol. To the best of our knowledge, the CCSD(T)/aug-cc-pVTZ calculation for CH4@(H2O)20 is the largest one reported to date (168 valence electrons, 1978 basis functions and the correlation of 84 doubly occupied and 1873 virtual orbitals) and required a scalable implementation of the (T) module on 6144 nodes (350208 cores) of the “Cori” supercomputer at the National Energy Research Supercomputing Center (NERSC) for a total execution time of 195 minutes (for the (T) part). These efficient scalable implementations of highly correlated methods offer the capability to obtain long-lasting benchmarks of intermolecular interactions in complex systems. They also provide a path towards parametrizing classical potentials needed to study the dynamical and transport properties in these complex systems as well as assess the accuracy of lower scaling electronic structure methods such as Density Functional Theory (DFT) and MP2 including its spin-biased variants.

Heindel, Joseph↗
DOE OSTI2023

Barrier Heights for Diels–Alder Transition States Leading to Pentacyclic Adducts: A Benchmark Study of Crowded, Strained Transition States of Large Molecules

Theoretical characterization of reactions of complex molecules depends on providing consistent accuracy for the relative energies of intermediates and transition states. Here we employ the DLPNO-CCSD(T) method with core–valence correlation, large basis sets, and extrapolation to the CBS limit to provide benchmark values for Diels–Alder transition states leading to competitive strained pentacyclic adducts. We then used those benchmarks to test a diverse set of wave function and density functional methods for the absolute and relative barrier heights of these transition states. Our results show that only a few of the tested density functionals can predict the absolute barrier heights satisfactorily, although relative barrier heights are more accurate. The most accurate functionals tested are ωB97M-V, M11plus, ωB97X-V, PBE-D3(0), M11, and MN15 with MUDs from best estimates less than 3.0 kcal. Furthermore, these findings can guide selection of density functionals for future studies of crowded, strained transition states of large molecules.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Benchmarking quantum logic operations relative to thresholds for fault tolerance

Contemporary methods for benchmarking noisy quantum processors typically measure average error rates or process infidelities. However, thresholds for fault-tolerant quantum error correction are given in terms of worst-case error rates—defined via the diamond norm—which can differ from average error rates by orders of magnitude. One method for resolving this discrepancy is to randomize the physical implementation of quantum gates, using techniques like randomized compiling (RC). In this work, we use gate set tomography to perform precision characterization of a set of two-qubit logic gates to study RC on a superconducting quantum processor. We find that, under RC, gate errors are accurately described by a stochastic Pauli noise model without coherent errors, and that spatially correlated coherent errors and non-Markovian errors are strongly suppressed. We further show that the average and worst-case error rates are equal for randomly compiled gates, and measure a maximum worst-case error of 0.0197(3) for our gate set. Our results show that randomized benchmarks are a viable route to both verifying that a quantum processor’s error rates are below a fault-tolerance threshold, and to bounding the failure rates of near-term algorithms, if—and only if—gates are implemented via randomization methods which tailor noise.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Quantum annealing for combinatorial optimization: a benchmarking study

Quantum annealing (QA) has the potential to significantly improve solution quality and reduce time complexity in solving combinatorial optimization problems compared to classical optimization methods. However, due to the limited number of qubits and their connectivity, the QA hardware did not show such an advantage over classical methods in past benchmarking studies. Recent advancements in QA with more than 5000 qubits, enhanced qubit connectivity, and the hybrid architecture promise to realize the quantum advantage. Here, we use a quantum annealer with state-of-the-art techniques and benchmark its performance against classical solvers. To compare their performance, we solve over 50 optimization problem instances represented by large and dense Hamiltonian matrices using quantum and classical solvers. The results demonstrate that a state-of-the-art quantum solver has higher accuracy (~0.013%) and a significantly faster problem-solving time (~6561×) than the best classical solver. Our results highlight the advantages of leveraging QA over classical counterparts, particularly in hybrid configurations, for achieving high accuracy and substantially reduced problem solving time in large-scale real-world optimization problems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Benchmarking highly entangled states on a 60-atom analogue quantum simulator

Abstract Quantum systems have entered a competitive regime in which classical computers must make approximations to represent highly entangled quantum states 1,2 . However, in this beyond-classically-exact regime, fidelity comparisons between quantum and classical systems have so far been limited to digital quantum devices 2–5 , and it remains unsolved how to estimate the actual entanglement content of experiments 6 . Here, we perform fidelity benchmarking and mixed-state entanglement estimation with a 60-atom analogue Rydberg quantum simulator, reaching a high-entanglement entropy regime in which exact classical simulation becomes impractical. Our benchmarking protocol involves extrapolation from comparisons against an approximate classical algorithm, introduced here, with varying entanglement limits. We then develop and demonstrate an estimator of the experimental mixed-state entanglement 6 , finding our experiment is competitive with state-of-the-art digital quantum devices performing random circuit evolution 2–5 . Finally, we compare the experimental fidelity against that achieved by various approximate classical algorithms, and find that only the algorithm we introduce is able to keep pace with the experiment on the classical hardware we use. Our results enable a new model for evaluating the ability of both analogue and digital quantum devices to generate entanglement in the beyond-classically-exact regime, and highlight the evolving divide between quantum and classical systems.

Science & Technology - Other Topics↗
DOE OSTI2023

BigNeuron: a resource to benchmark and predict performance of algorithms for automated tracing of neurons in light microscopy datasets

BigNeuron is an open community bench-testing platform with the goal of setting open standards for accurate and fast automatic neuron tracing. We gathered a diverse set of image volumes across several species that is representative of the data obtained in many neuroscience laboratories interested in neuron tracing. Here, we report generated gold standard manual annotations for a subset of the available imaging datasets and quantified tracing quality for 35 automatic tracing algorithms. The goal of generating such a hand-curated diverse dataset is to advance the development of tracing algorithms and enable generalizable benchmarking. Together with image quality features, we pooled the data in an interactive web application that enables users and developers to perform principal component analysis, t-distributed stochastic neighbor embedding, correlation and clustering, visualization of imaging and tracing data, and benchmarking of automatic tracing algorithms in user-defined data subsets. The image quality metrics explain most of the variance in the data, followed by neuromorphological features related to neuron size. Furthermore, we observed that diverse algorithms can provide complementary information to obtain accurate results and developed a method to iteratively combine methods and generate consensus reconstructions. The consensus trees obtained provide estimates of the neuron structure ground truth that typically outperform single algorithms in noisy datasets. However, specific algorithms may outperform the consensus tree strategy in specific imaging conditions. Finally, to aid users in predicting the most accurate automatic tracing results without manual annotations for comparison, we used support vector machine regression to predict reconstruction quality given an image volume and a set of automatic tracings.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Energy landscapes from cryo-EM snapshots: a benchmarking study

Abstract Biomolecules undergo continuous conformational motions, a subset of which are functionally relevant. Understanding, and ultimately controlling biomolecular function are predicated on the ability to map continuous conformational motions, and identify the functionally relevant conformational trajectories. For equilibrium and near-equilibrium processes, function proceeds along minimum-energy pathways on one or more energy landscapes, because higher-energy conformations are only weakly occupied. With the growing interest in identifying functional trajectories, the need for reliable mapping of energy landscapes has become paramount. In response, various data-analytical tools for determining structural variability are emerging. A key question concerns the veracity with which each data-analytical tool can extract functionally relevant conformational trajectories from a collection of single-particle cryo-EM snapshots. Using synthetic data as an independently known ground truth, we benchmark the ability of four leading algorithms to determine biomolecular energy landscapes and identify the functionally relevant conformational paths on these landscapes. Such benchmarking is essential for systematic progress toward atomic-level movies of continuous biomolecular function.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Federated benchmarking of medical artificial intelligence with MedPerf

Medical artificial intelligence (AI) has tremendous potential to advance healthcare by supporting and contributing to the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving both healthcare provider and patient experience. Unlocking this potential requires systematic, quantitative evaluation of the performance of medical AI models on large-scale, heterogeneous data capturing diverse patient populations. Here, to meet this need, we introduce MedPerf, an open platform for benchmarking AI models in the medical domain. MedPerf focuses on enabling federated evaluation of AI models, by securely distributing them to different facilities, such as healthcare organizations. This process of bringing the model to the data empowers each facility to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status and real-world deployment, our roadmap and, importantly, the use of MedPerf with multiple international institutions within cloud-based technology and on-premises scenarios. Finally, we welcome new contributions by researchers and organizations to further strengthen MedPerf as an open benchmarking platform.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2026

A curated benchmark for cofolding models on kinase conformational states

Abstract Protein kinases are critical drug targets, requiring therapeutics that can modulate their active and inactive conformational states. While cofolding models can generate global folds directly from kinase sequences and ligand SMILES strings, these models have not yet been tested on their ability to recover ligand-induced-fit conformational states of the kinase proteins. Here, we introduce KinConfBench, a curated benchmark of 2225 high-quality human kinase chains to evaluate the ability of four state-of-the-art cofolding models—Boltz-2, Chai-1, Protenix, and RoseTTAFold-All-Atom—to recover both canonical and rare conformational states. We show that geometric success metrics of a ligand pose in the active site do not correlate strongly with the correct kinase conformational state, motivating a new set of dynamical benchmarks for assessing cofolding models. While all four cofolding models achieve ~60–80% prediction accuracy for kinase conformational classification, they exhibit severe mode collapse when performing multiple inferences, show negligible structural diversity in sampling induced-fit motions, and display a prevalent “apo-drift” in which most cofolding models predominantly predict the kinase to be in its ligand-free state. Our results highlight that capturing ligand-induced protein conformational diversity, not just geometric fit, is critical for next-generation structure-based drug discovery.

Sun, Kunyang↗
DOE OSTI2020

Axisymmetric simulations of vertical displacement events in tokamaks: A benchmark of M3D-C 1 , NIMROD, and JOREK

A benchmark exercise for the modeling of vertical displacement events (VDEs) is presented and applied to the 3D nonlinear magnetohydrodynamic codes M3D-C 1 , JOREK, and NIMROD. The simulations are based on a vertically unstable NSTX equilibrium enclosed by an axisymmetric resistive wall with a rectangular cross section. A linear dependence of the linear VDE growth rates on the resistivity of the wall is recovered for sufficiently large wall conductivity and small temperatures in the open field line region. We report the benchmark results show good agreement between the VDE growth rates obtained from linear NIMROD and M3D-C 1 simulations and from the linear phase of axisymmetric nonlinear JOREK, NIMROD, and M3D-C 1 simulations. Axisymmetric nonlinear simulations of a full VDE performed with the three codes are compared, and an excellent agreement is found regarding the plasma location and plasma currents, as well as eddy and halo currents in the wall.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2020

Solvation at metal/water interfaces: An ab initio molecular dynamics benchmark of common computational approaches

Determining the influence of the solvent on electrochemical reaction energetics is a central challenge in our understanding of electrochemical interfaces. To date, it is unclear how well existing methods predict solvation energies at solid/liquid interfaces, since they cannot be assessed experimentally. Ab initio molecular dynamics (AIMD) simulations present a physically highly accurate, but also a very costly approach. In this work, we employ extensive AIMD simulations to benchmark solvation at charge-neutral metal/water interfaces against commonly applied continuum solvent models. We consider a variety of adsorbates including *CO, *CHO, *COH, *OCCHO, *OH, and *OOH on Cu, Au, and Pt facets solvated by water. The surfaces and adsorbates considered are relevant, among other reactions, to electrochemical CO 2 reduction and the oxygen redox reactions. We determine directional hydrogen bonds and steric water competition to be critical for a correct description of solvation at the metal/water interfaces. As a consequence, we find that the most frequently applied continuum solvation methods, which do not yet capture these properties, do not presently provide more accurate energetics over simulations in vacuum. We find most of the computed benchmark solvation energies to linearly scale with hydrogen bonding or competitive water adsorption, which strongly differ across surfaces. Furthermore, we determine solvation energies of adsorbates to be non-transferable between metal surfaces, in contrast to standard practice.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

Gyrokinetic benchmark of the electron temperature-gradient instability in the pedestal region

Transport from turbulence driven by the electron temperature-gradient (ETG) instability is likely a major source of electron heat losses through the pedestal. Due to extreme gradients and strong shaping, ETG instabilities in the pedestal are distinct from those in the core, having, for example, multiple branches (toroidal and slab) in different wavenumber ranges. Due to its importance for pedestal transport, and its rather exotic character, a rigorous multi-code benchmarking exercise is imperative. In this work, we describe such an exercise, wherein we have carried out a detailed comparison of local linear pedestal ETG simulations using three gyrokinetic codes, CGYRO, GEM, and GENE and testing different geometric parameters (such as circular, Miller, and equilibrium EFIT geometry). The resulting linear frequencies, growth rates, and eigenfunctions show very good agreement between the codes in the three types of employed geometries. A nonlinear benchmark between CGYRO and GENE is also described, exhibiting good agreement (a maximum of 20% difference in the heat fluxes computed) at two locations in the pedestal. This lays the foundation for confidently modeling ETG turbulence in the pedestal

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2021

NENCI-2021. I. A large benchmark database of non-equilibrium non-covalent interactions emphasizing close intermolecular contacts

In this work, we present NENCI-2021, a benchmark database of ~8000 Non-Equilibirum Non-Covalent Interaction energies for a large and diverse selection of intermolecular complexes of biological and chemical relevance. To meet the growing demand for large and high-quality quantum mechanical data in the chemical sciences, NENCI-2021 starts with the 101 molecular dimers in the widely used S66 and S101 databases and extends the scope of these works by (i) including 40 cation–π and anion–π complexes, a fundamentally important class of non-covalent interactions that are found throughout nature and pose a substantial challenge to theory, and (ii) systematically sampling all 141 intermolecular potential energy surfaces (PESs) by simultaneously varying the intermolecular distance and intermolecular angle in each dimer. Designed with an emphasis on close contacts, the complexes in NENCI-2021 were generated by sampling seven intermolecular distances along each PES (ranging from 0.7× to 1.1× the equilibrium separation) and nine intermolecular angles per distance (five for each ion–π complex), yielding an extensive database of 7763 benchmark intermolecular interaction energies (E int ) obtained at the coupled-cluster with singles, doubles, and perturbative triples/complete basis set [CCSD(T)/CBS] level of theory. The E int values in NENCI-2021 span a total of 225.3 kcal/mol, ranging from -38.5 to +186.8 kcal/mol, with a mean (median) E int value of -1.06 kcal/mol (-2.39 kcal/mol). In addition, a wide range of intermolecular atom-pair distances are also present in NENCI-2021, where close intermolecular contacts involving atoms that are located within the so-called van der Waals envelope are prevalent—these interactions, in particular, pose an enormous challenge for molecular modeling and are observed in many important chemical and biological systems. A detailed symmetry-adapted perturbation theory (SAPT)- based energy decomposition analysis also confirms the diverse and comprehensive nature of the intermolecular binding motifs present in NENCI-2021, which now includes a significant number of primarily induction-bound dimers (e.g., cation–π complexes). NENCI-2021 thus spans all regions of the SAPT ternary diagram, thereby warranting a new four-category classification scheme that includes complexes primarily bound by electrostatics (3499), induction (700), dispersion (1372), or mixtures thereof (2192). A critical error analysis performed on a representative set of intermolecular complexes in NENCI-2021 demonstrates that the E int values provided herein have an average error of ±0.1 kcal/mol, even for complexes with strongly repulsive E int values, and maximum errors of ±0.2–0.3 kcal/mol (i.e., ~±1.0 kJ/mol) for the most challenging cases. For these reasons, we expect that NENCI-2021 will play an important role in the testing, training, and development of next-generation classical and polarizable force fields, density functional theory approximations, wavefunction theory methods, and machine learning based intra- and inter-molecular potentials.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

Preliminary benchmarks and analysis of boundary conditions in a trenched microstructured silicon radiation detector

Microstructured neutron detectors have the benefit of enhanced neutron detection efficiency as compared to planar devices, achieved by etching 6 LiF-filled trenches on the top surface of a silicon PIN diode. This sensor geometry results in a complex electric field distribution and depletion characteristics within the diode under reverse bias. For the first time on record, the effects of a fixed oxide charge on the microstructured device depletion characteristics and mobile carrier transport is investigated. Prototype detectors were fabricated with non-conformal surface doping. Capacitance voltage and current voltage measurements were performed for these prototypes and compared with COMSOL Multiphysics simulations. A spectral response from an 241Am alpha particle source was acquired and analyzed. It was found that monoenergetic alpha particles produce three prominent peaks in the pulse height spectrum output by the device. The peaks were confirmed by simulations to correlate with dead layers and incident trajectories into the microstructure. It was also found that significant differences in pulse rise time result, corresponding with events arriving in a low-field region in the fins and a high-field region in the bulk. Geant4 was utilized for radiation transport, interaction modeling, and benchmarking the spectral data. The results of this simulation work provide confidence in the ability to attain and benchmark electrical characteristics and spectral data for semiconductor radiation detectors employing complex microstructures.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗