Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmark”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,117 records · Page 62
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Evaluation of Ocean Biogeochemistry and Carbon Cycling in CMIP Earth System Models With the International Ocean Model Benchmarking (IOMB) Software System

Abstract The International Ocean Model Benchmarking (IOMB) software package is a new community resource that we use here to evaluate surface and upper ocean biogeochemical variables and integrated anthropogenic carbon uptake from earth system models (ESMs) contributing to the 5th and 6th phases of the Coupled Model Intercomparison Project (CMIP5 and CMIP6). IOMB generates graphics and tables for systematically comparing model predictions against multiple datasets. Our analysis reveals some improvement in the multi‐model mean from CMIP5 to CMIP6 for most of the variables we examined. Compared to data‐constrained estimates of ocean anthropogenic carbon uptake for the 1994–2007 period, negative biases exist for many models between 30 and 50°S. Global model estimates of anthropogenic carbon uptake for the same period do not change significantly from CMIP5 to CMIP6, with the combined ensemble mean estimate of 27.8 ± 0.5 Pg C lower than a data‐constrained estimate of 33.0 ± 4.0 Pg C. At the same time, the change in the natural carbon inventory from CMIP is estimated to be a source of 0.7 ± 0.3 Pg C, which is considerably smaller in magnitude than a data‐constrained estimate of 5.0 ± 3.0 Pg C. With chlorofluorocarbon (CFC) predictions available for several models, we demonstrate that negative anthropogenic dissolved inorganic carbon biases coincide with negative biases in CFC concentration, highlighting the importance of weak exchange between the surface and interior ocean in regulating rates of anthropogenic carbon uptake. To examine the robustness of this attribution across the CMIP models, we calculate the global vertical temperature gradient between 200 and 1,000 m as a metric for global stratification and exchange between the surface and deeper waters. We find a linear relationship between the bias of the vertical temperature gradients and the bias in global anthropogenic carbon uptake, consistent with the hypothesis that model biases in anthropogenic carbon uptake are related to biases in surface‐to‐interior exchange by physical processes.

58 GEOSCIENCES↗
DOE OSTI2026

Rates of Sea‐Level Rise Are Highly Sensitive to Ice Viscosity Parameters in Model Benchmarks

Glacier flow plays a major role in current and future rates of globally averaged sea-level rise. The viscosity of glacial ice, controlling the rate of flow, decreases as stress increases and is highly sensitive to the value of the stress exponent, $n$, in the constitutive equation for viscous flow. Glaciologists and climate modelers almost exclusively assume $n=3$ when modeling ice flow and projecting sea-level rise through forward modeling. However, recent work suggests that $n\approx 4$ better fits observations, prompting the question: How sensitive are projections of sea-level rise to the value of $n$? We use an established community ice flow model and standard benchmark experiments designed as an idealized representation of Pine Island Glacier, West Antarctica. While initializing an $n=3$ model to match observations of an $n=4$ ice sheet is possible, we find that incorrectly assuming $n=3$ when in fact $n=4$ dramatically underestimates rates of sea-level rise. The scale of this error grows nonlinearly with the magnitude of the climate forcing, acting to increase projection uncertainties. Additionally, we find that models often account for this stress-dependent rheology mismatch during model initialization in a way that masks this rheological effect in the short term while leaving model outputs vulnerable to larger biases in longer-term projections. Initializations to observations of Pine Island Glacier display similar rheology-mismatch fingerprints to our idealized example.

climate sensitivity↗
DOE OSTI2026

Systematic benchmarking demonstrates large language models have not reached the diagnostic accuracy of traditional rare-disease decision support tools

Large language models (LLMs) show promise in supporting differential diagnosis, but their performance is challenging to evaluate due to the unstructured nature of their responses, and their accuracy compared to existing diagnostic tools is not well characterized. To assess the current capabilities of LLMs to diagnose genetic diseases, we benchmarked these models on 5213 previously published case reports using the Phenopacket Schema, the Human Phenotype Ontology and Mondo disease ontology. Prompts generated from each phenopacket were sent to seven LLMs, including four generalist models and three LLMs specialized for medical applications. The same phenopackets were used as input to a widely used diagnostic tool, Exomiser, in phenotype-only mode. The best LLM ranked the correct diagnosis first in 23.6% of cases, whereas Exomiser did so in 35.5% of cases. While the performance of LLMs for supporting differential diagnosis has been improving, it has not reached the level of commonly used traditional bioinformatics tools. Future research is needed to determine the best approach to incorporate LLMs into diagnostic pipelines.

Reese, Justin T. [Lawrence Berkeley National Labor↗
DOE OSTI2021

Benchmarking graph neural networks for materials chemistry

Graph neural networks (GNNs) have received intense interest as a rapidly expanding class of machine learning models remarkably well-suited for materials applications. To date, a number of successful GNNs have been proposed and demonstrated for systems ranging from crystal stability to electronic property prediction and to surface chemistry and heterogeneous catalysis. However, a consistent benchmark of these models remains lacking, hindering the development and consistent evaluation of new models in the materials field. Here, we present a workflow and testing platform, MatDeepLearn, for quickly and reproducibly assessing and comparing GNNs and other machine learning models. We use this platform to optimize and evaluate a selection of top performing GNNs on several representative datasets in computational materials chemistry. From our investigations we note the importance of hyperparameter selection and find roughly similar performances for the top models once optimized. We identify several strengths in GNNs over conventional models in cases with compositionally diverse datasets and in its overall flexibility with respect to inputs, due to learned rather than defined representations. Meanwhile several weaknesses of GNNs are also observed including high data requirements, and suggestions for further improvement for applications in materials chemistry are discussed.

36 MATERIALS SCIENCE↗
DOE OSTI2023

High-throughput calculations of charged point defect properties with semi-local density functional theory—performance benchmarks for materials screening applications

Abstract Calculations of point defect energetics with Density Functional Theory (DFT) can provide valuable insight into several optoelectronic, thermodynamic, and kinetic properties. These calculations commonly use methods ranging from semi-local functionals with a-posteriori corrections to more computationally intensive hybrid functional approaches. For applications of DFT-based high-throughput computation for data-driven materials discovery, point defect properties are of interest, yet are currently excluded from available materials databases. This work presents a benchmark analysis of automated, semi-local point defect calculations with a-posteriori corrections, compared to 245 “gold standard” hybrid calculations previously published. We consider three different a-posteriori correction sets implemented in an automated workflow, and evaluate the qualitative and quantitative differences among four different categories of defect information: thermodynamic transition levels, formation energies, Fermi levels, and dopability limits. We highlight qualitative information that can be extracted from high-throughput calculations based on semi-local DFT methods, while also demonstrating the limits of quantitative accuracy.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Downfolding from ab initio to interacting model Hamiltonians: comprehensive analysis and benchmarking of the DFT+cRPA approach

Abstract Model Hamiltonians are regularly derived from first principles to describe correlated matter. However, the standard methods for this contain a number of largely unexplored approximations. For a strongly correlated impurity model system, here we carefully compare a standard downfolding technique with the best possible ground-truth estimates for charge-neutral excited-state energies and wave functions using state-of-the-art first-principles many-body wave function approaches. To this end, we use the vanadocene molecule and analyze all downfolding aspects, including the Hamiltonian form, target basis, double-counting correction, and Coulomb interaction screening models. We find that the choice of target-space basis functions emerges as a key factor for the quality of the downfolded results, while orbital-dependent double-counting corrections diminish the quality. Background screening of the Coulomb interaction matrix elements primarily affects crystal-field excitations. Our benchmark uncovers the relative importance of each downfolding step and offers insights into the potential accuracy of minimal downfolded model Hamiltonians.

Chemistry↗
DOE OSTI2022

A quantum hamiltonian simulation benchmark

Hamiltonian simulation is one of the most important problems in quantum computation, and quantum singular value transformation (QSVT) is an efficient way to simulate a general class of Hamiltonians. However, the QSVT circuit typically involves multiple ancilla qubits and multi-qubit control gates. In order to simulate a certain class of n-qubit random Hamiltonians, we propose a drastically simplified quantum circuit that we refer to as the minimal QSVT circuit, which uses only one ancilla qubit and no multi-qubit controlled gates. We formulate a simple metric called the quantum unitary evolution score (QUES), which is a scalable quantum benchmark and can be verified without any need for classical computation. Under the globally depolarized noise model, we demonstrate that QUES is directly related to the circuit fidelity, and the potential classical hardness of an associated quantum circuit sampling problem. Under the same assumption, theoretical analysis suggests there exists an ‘optimal’ simulation time t opt ≈ 4.81, at which even a noisy quantum device may be sufficient to demonstrate the potential classical hardness.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Best practices and benchmarks for intact protein analysis for top-down mass spectrometry

One gene can give rise to many functionally distinct proteoforms, each of which has a characteristic molecular mass. Many laboratories, however, forgo the challenging mass analysis of intact proteins. Instead, these researchers resort to well-established methods for proteolytic digests. In this work, the Consortium for Top-Down Proteomics (CTDP) addresses the unmet need for robust protocols for obtaining intact protein mass spectra. We quantify and rationalize the signal suppression associated with common interfering substances. A decision tree, based upon sample composition and experimental goals, guides researchers to a given protocol. Cross-platform analytical benchmarks are provided to allow users to gauge their proficiency and to promote method standardization. This work should permit users at any experience level to obtain high-quality intact mass spectra of native and denatured proteins from mixtures of varying complexity, including intact antibodies and membrane proteins.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

A multi-scale time-series dataset with benchmark for machine learning in decarbonized energy grids

The electric grid is a key enabling infrastructure for the ambitious transition towards carbon neutrality as we grapple with climate change. With deepening penetration of renewable resources, the reliable operation of the electric grid becomes increasingly challenging. In this paper, we present PSML, a first-of-its-kind open-access multi-scale time-series dataset, to aid in the development of data-driven machine learning (ML)-based approaches towards reliable operation of future electric grids. The dataset is synthesized from a joint transmission and distribution electric grid to capture the increasingly important interactions and uncertainties of the grid dynamics, containing power, voltage and current measurements over multiple spatio-temporal scales. Using PSML, we provide state-of-the-art ML benchmarks on three challenging use cases of critical importance to achieve: (i) early detection, accurate classification and localization of dynamic disturbances; (ii) robust hierarchical forecasting of load and renewable energy; and (iii) realistic synthetic generation of physical-law-constrained measurements. We envision that this dataset will provide use-inspired ML research in safety-critical systems, while simultaneously enabling ML researchers to contribute towards decarbonization of energy sectors.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Benchmarking second and third-generation sequencing platforms for microbial metagenomics

Shotgun metagenomic sequencing is a common approach for studying the taxonomic diversity and metabolic potential of complex microbial communities. Current methods primarily use second generation short read sequencing, yet advances in third generation long read technologies provide opportunities to overcome some of the limitations of short read sequencing. Here, we compared seven platforms, encompassing second generation sequencers (Illumina HiSeq 300, MGI DNBSEQ-G400 and DNBSEQ-T7, ThermoFisher Ion GeneStudio S5 and Ion Proton P1) and third generation sequencers (Oxford Nanopore Technologies MinION R9 and Pacific Biosciences Sequel II). We constructed three uneven synthetic microbial communities composed of up to 87 genomic microbial strains DNAs per mock, spanning 29 bacterial and archaeal phyla, and representing the most complex and diverse synthetic communities used for sequencing technology comparisons. Our results demonstrate that third generation sequencing have advantages over second generation platforms in analyzing complex microbial communities, but require careful sequencing library preparation for optimal quantitative metagenomic analysis. Our sequencing data also provides a valuable resource for testing and benchmarking bioinformatics software for metagenomics.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Increasing the hardness of posiform planting using random QUBOs for programmable quantum annealer benchmarking

Posiform planting is a method for constructing QUBO instances with a unique planted solution that can be tailored to arbitrary connectivity graphs. In this study we investigate making posiform planted QUBOs computationally harder by fusing many smaller random Ising models, whose global minimum is computed classically, with posiform planted QUBOs. The unique ground state of the resulting QUBO is the concatenation of (exactly one of) the ground states of each smaller problem. Our method generates QUBO instances that have a unique solution, are native to the hardware graph, and have tunable computational hardness. We use our QUBOs to benchmark three D-Wave quantum annealing processors (with 563–5627 qubits), and compare them against simulated annealing and Gurobi. Surprisingly, we find that the D-Wave ground state sampling success rate is not dependent on the glued random QUBO size, and that some QUBO classes are solved at high success rates at short annealing times on the Zephyr processors.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2018

Unified field studies of the algae testbed public-private partnership as the benchmark for algae agronomics

National scale agronomic projections are an important input for assessing potential benefits of algae cultivation on the future of innovative agriculture. The Algae Testbed Public-Private Partnership was established with the goal of investigating open pond algae cultivation across different geographic, climatic, seasonal, and operational conditions while setting the benchmark for quality data collection, analysis, and dissemination. Identical algae cultivation systems and data analysis methodologies were established at testbed sites across the continental United States and Hawaii. Within this framework, the Unified Field Studies were designed for algae cultivation during all 4 seasons across the testbed network. With increasingly diverse algae research and development, and field deployment strategies, the challenges associated with data collection, quality, and dissemination increase dramatically. The dataset presented here is the complete, curated, climatic, cultivation, harvest, and biomass composition data for each season at each site. Furthermore, these data enable others to do in-depth cultivation, harvest, techno-economic, life cycle, resource, and predictive growth modelling analysis, as well as development of crop protection strategies throughout the algae cultivation industry.

09 BIOMASS FUELS↗
DOE OSTI2013

Functionalized Single-Walled Carbon Nanotube-Based Fuel Cell Benchmarked Against US DOE 2017 Technical Targets

Chemically modified single-walled carbon nanotubes (SWNTs) with varying degrees of functionalization were utilized for the fabrication of SWNT thin film catalyst support layers (CSLs) in polymer electrolyte membrane fuel cells (PEMFCs), which were suitable for benchmarking against the US DOE 2017 targets. Use of the optimum level of SWNT -COOH functionality allowed the construction of a prototype SWNT-based PEMFC with total Pt loading of 0.06 mg Pt /cm 2 - well below the value of 0.125 mg Pt /cm 2 set as the US DOE 2017 technical target for total Pt group metals (PGM) loading. This prototype PEMFC also approaches the technical target for the total Pt content per kW of power (<0.125 g PGM /kW) at cell potential 0.65 V: a value of 0.15 g Pt /kW was achieved at 80°C/22 psig testing conditions, which was further reduced to 0.12 g Pt /kW at 35 psig back pressure.

77 NANOSCIENCE AND NANOTECHNOLOGY↗
DOE OSTI2013

Copper doped TiO2 nanoparticles characterized by X-ray absorption spectroscopy, total scattering, and powder diffraction – a benchmark structure–property study

Metal functionalized nanoparticles potentially have improved properties e.g. in catalytic applications, but their precise structures are often very challenging to determine. Here we report a structural benchmark study based on tetragonal anatase TiO 2 nanoparticles containing 0–2 wt% copper. The particles were synthesized by continuous flow synthesis under supercritical water–isopropanol conditions. Size determination using synchrotron PXRD, TEM, and X-ray total scattering reveals 5–7 nm monodisperse particles. The precise dopant structure and thermal stability of the highly crystalline powders were characterized by X-ray absorption spectroscopy and multi-temperature synchrotron PXRD (300–1000 K). The combined evidence reveals that copper is present as a dopant on the particle surfaces, most likely in an amorphous oxide or hydroxide shell. UV-VIS spectroscopy shows that copper presence at concentrations higher than 0.3 wt% lowers the band gap energy. The particles are unaffected by heating to 600 K, while growth and partial transformation to rutile TiO 2 occur at higher temperatures. Anisotropic unit cell behavior of anatase is observed as a consequence of the particle growth (a decreases and c increases).

Lock, Nina↗
DOE OSTI2015

Benchmarking density functional theory predictions of framework structures and properties in a chemically diverse test set of metal-organic frameworks

We compiled a test set of chemically and topologically diverse Metal–Organic Frameworks (MOFs) with high accuracy experimentally derived crystallographic structure data. The test set was used to benchmark the performance of Density Functional Theory (DFT) functionals (M06L, PBE, PW91, PBE-D2, PBE-D3, and vdW-DF2) for predicting lattice parameters, unit cell volume, bonded parameters and pore descriptors. On average PBE-D2, PBE-D3, and vdW-DF2 predict more accurate structures, but all functionals predicted pore diameters within 0.5 Å of the experimental diameter for every MOF in the test set. The test set was also used to assess the variance in performance of DFT functionals for elastic properties and atomic partial charges. The DFT predicted elastic properties such as minimum shear modulus and Young's modulus can differ by an average of 3 and 9 GPa for rigid MOFs such as those in the test set. Moreover, we calculated the partial charges by vdW-DF2 deviate the most from other functionals while there is no significant difference between the partial charges calculated by M06L, PBE, PW91, PBE-D2 and PBE-D3 for the MOFs in the test set. We find that while there are differences in the magnitude of the properties predicted by the various functionals, these discrepancies are small compared to the accuracy necessary for most practical applications.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Inverse deep learning methods and benchmarks for artificial electromagnetic material design

In this work we investigate the use of deep inverse models (DIMs) for designing artificial electromagnetic materials (AEMs) – such as metamaterials, photonic crystals, and plasmonics – to achieve some desired scattering properties (e.g., transmission or reflection spectrum). DIMs are deep neural networks (i.e., deep learning models) that are specially-designed to solve ill-posed inverse problems. There has recently been tremendous growth in the use of DIMs for solving AEM design problems however there has been little comparison of these approaches to examine their absolute and relative performance capabilities. In this work we compare eight state-of-the-art DIMs on three unique AEM design problems, including two models that are novel to the AEM community. Our results indicate that DIMs can rapidly produce accurate designs to achieve a custom desired scattering on all three problems. Although no single model always performs best, the Neural-Adjoint approach achieves the best overall performance across all problem settings. As a final contribution we show that not all AEM design problems are ill-posed, and in such cases a conventional deep neural network can perform better than DIMs. We recommend that a deep neural network is always employed as a simple baseline approach when addressing AEM design problems. Furthermore, we publish python code for our AEM simulators and our DIMs to enable easy replication of our results, and benchmarking of new DIMs by the AEM community.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

Benchmark relativistic delta-coupled-cluster calculations of K-edge core-ionization energies of third-row elements

Here a benchmark computational study of K-edge core-ionization energies of third-row elements using relativistic delta-coupled-cluster (ΔCC) methods and a revised core-valence separation (CVS) scheme is reported. High-level relativistic (HLR) corrections beyond the spin-free exact two-component theory in its one-electron variant (SFX2C-1e), including the contributions from two-electron picture-change effects, spin–orbit coupling, the Breit term, and quantum electrodynamics effects, have been taken into account and demonstrated to play an important role. Relativistic ΔCC calculations are shown to provide accurate results for core-ionization energies of third-row elements. The SFX2C-1e-CVS-ΔCC results augmented with HLR corrections show a maximum deviation of less than 0.5 eV with respect to experimental values.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗