Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmarking”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding

Quantum computing calibration depends on interpreting experimental data, and calibration plots provide the most universal human-readable representation for this task, yet no systematic evaluation exists of how well vision-language models (VLMs) interpret them. We introduce QCalEval, the first VLM benchmark for quantum calibration plots: 243 samples across 87 scenario types from 22 experiment families, spanning superconducting qubits and neutral atoms, evaluated on six question types in both zero-shot and in-context learning settings. The best general-purpose zero-shot model reaches a mean score of 72.3, and many open-weight models degrade under multi-image in-context learning, whereas frontier closed models improve substantially. A supervised fine-tuning ablation at the 9-billion-parameter scale shows that SFT improves zero-shot performance but cannot close the multimodal in-context learning gap. As a reference case study, we release NVIDIA Ising Calibration 1, an open-weight model based on Qwen3.5-35B-A3B that reaches 74.7 zero-shot average score.

Cao, Shuxiang↗
DOE OSTI2024

Recent Advances in Immobilizing and Benchmarking Molecular Catalysts for Artificial Photosynthesis

Transition metal complexes have been widely used as catalysts or chromophores in artificial photosynthesis. Traditionally, they are employed in homogeneous settings. Despite their functional versatility and structural tunability, broad industrial applications of these catalysts are impeded by the limitations of homogeneous catalysis such as poor catalyst recyclability, solvent constraints (mostly organic solvents), and catalyst durability. Over the past few decades, researchers have developed various methods for molecular catalyst heterogenization to overcome these limitations. Here, in this review, we summarize recent developments in heterogenization strategies, with a focus on describing methods employed in the heterogenization process and their effects on catalytic performances. Alongside the in-depth discussion of heterogenization strategies, this review aims to provide a concise overview of the key metrics associated with heterogenized systems. We hope this review will aid researchers who are new to this research field in gaining a better understanding.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

A Practical Probabilistic Benchmark for AI Weather Models

Since the weather is chaotic, it is necessary to forecast an ensemble of future states. Recently, multiple AI weather models have emerged claiming breakthroughs in deterministic skill. Unfortunately, it is hard to fairly compare ensembles of AI forecasts because variations in ensembling methodology become confounding and the baseline data volume is immense. We address this by scoring lagged initial condition ensembles—whereby an ensemble can be constructed from a library of deterministic hindcasts. This allows the first parameter‐free intercomparison of leading AI weather models' probabilistic skill against an operational baseline. Lagged ensembles of the two leading AI weather models, GraphCast and Pangu, perform similarly even though the former outperforms the latter in deterministic scoring. These results are elaborated upon by sensitivity tests showing that commonly used multiple time‐step loss functions damage ensemble calibration.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Electronic structure analysis and DFT benchmarking of Rydberg-type alkali-metal-crown ether, -cryptand, and -adamanzane complexes

Density functional theory (DFT) and electron propagator theory (EPT) calculations were performed to study ground and excited electronic structures of alkali-metal (M) coordinated 9-crown-3, 24-crown-8, [2.1.1]cryptand, o-Me 2 -1.1.1, and 3 6 Adamanzane complexes. Each complex bears an expanded electron in the periphery and occupies diffuse 1p-, 1d-, 1f-type molecular orbitals (or superatomic 1P, 1D, 1F orbitals) in excited electronic states. The calculated superatomic shell model of the M(9-crown-3) 2 is 1S, 1P, 1D, 1F, 2S, 2P, 2D, 1G and it is held by all other complexes up to the studied 1F level. Due to the highly diffuse nature of the electron, the ionization energies of these complexes are significantly lower (1.6–2.0 eV) and hence these complexes belong to the superalkali category. Further, the ab initio EPT ionization energy and the excitation energies of the Li(9-crown-3) 2 were used to evaluate DFT errors associated with a series of exchange correlation functionals that span multiple rungs of Jacob's ladder (i.e., GGA, meta-GGA, global GGA hybrid, meta-GGA hybrid, range-separated hybrid, double-hybrid). Among these, the best performing functional is the range-separated hybrid CAM-B3LYP and the errors are within 6% of high-level ab initio EPT results. The accuracy of CAM-B3LYP is indeed transferable to similar complexes and hence the findings are expected to accelerate the progression of studies of Rydberg-type systems.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

Benchmarking of X‐Ray Fluorescence Microscopy with Ion Beam Implanted Samples Showing Detection Sensitivity of Hundreds of Atoms

Abstract Single impurities in insulators are now often used for quantum sensors and single photon sources, while nanoscale semiconductor doping features are being constructed for electrical contacts in quantum technology devices, implying that new methods for sensitive, non‐destructive imaging of single‐ or few‐atom structures are needed. X‐ray fluorescence (XRF) can provide nanoscale imaging with chemical specificity, and features comprising as few as 100 000 atoms have been detected without any need for specialized or destructive sample preparation. Presently, the ultimate limits of sensitivity of XRF are unknown – here, gallium dopants in silicon are investigated using a high brilliance, synchrotron source collimated to a small spot. It is demonstrated that with a single‐pixel integration time of 1 s, the sensitivity is sufficient to identify a single isolated feature of only 3000 Ga impurities (a mass of just 350 zg). With increased integration (25 s), 650 impurities can be detected. The results are quantified using a calibration sample consisting of precisely controlled numbers of implanted atoms in nanometer‐sized structures. The results show that such features can now be mapped quantitatively when calibration samples are used, and suggest that, in the near future, planned upgrades to XRF facilities might achieve single‐atom sensitivity.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Benchmarking thermal energy storage cost for industrial process heat

Process heat accounts for roughly half of industrial energy demand, and currently 95% of process heat is derived from the combustion of natural gas, oil, and coal. Electrification of industrial heating could be an alternative, potentially expanding locations suitable for manufacturing; however, industrial facility owners may desire energy storage to stabilize energy costs. In this work, the economic benefits of pairing thermal storage with electrified process heat to reduce the average price paid for energy are analyzed. Cost savings focus on energy arbitrage, or leveraging flexible energy pricing schemes, alone. The cost of natural gas combustion across decades (2019-2060) is compared to the costs of electricity and thermal energy storage in four United States Independent System Operator (ISO) regions. Systems installed today may not yield positive net present value (NPV) compared to the use of natural gas. However, using estimated electricity prices, systems installed in 2030 using arbitrage alone could be profitable when compared to natural gas in some regions of the U.S. Furthermore, if capital expenditures could be reduced by 50% for sensible thermal storage systems by 2030, profitable systems are found across all regions. This implies that electrification of industrial process heat, when paired with inexpensive thermal energy storage systems, could be less expensive than brownfield natural gas systems, using arbitrage as the only source of revenue and without a dependency on any future policy drivers such as pricing externalities that could further incentivize the electrification of industrial process heat.

24 POWER TRANSMISSION AND DISTRIBUTION↗