Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmark”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Benchmarking the performance of a high-Q cavity qudit using random unitaries

High-coherence cavity resonators are excellent resources for encoding quantum information in higher-dimensional Hilbert spaces, moving beyond traditional qubit-based platforms. A natural strategy is to use the Fock basis to encode information in qudits. One can perform quantum operations on the cavity mode qudit by coupling the system to a non-linear ancillary transmon qubit. However, the performance of the cavity-transmon device is limited by the noisy transmons. It is, therefore, important to develop practical benchmarking tools for these qudit systems in an algorithm-agnostic manner. We gauge the performance of these qudit platforms using sampling tests such as the heavy output generation test as well as the linear cross-entropy benchmark, by way of simulations of such a system subject to realistic dominant noise channels. We use selective number-dependent arbitrary phase and unconditional displacement gates as our universal gateset. Our results show that contemporary transmons comfortably enable controlling a few tens of Fock levels of a cavity mode. This framework allows benchmarking even higher dimensional qudits as those become accessible with improved transmons.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2021

Benchmarking SARS CoV-2 Infection in the Workplace to Support Continuity of Operations

Objective: The COVID-19 pandemic jeopardizes continuity of operations of workplaces and the health and safety of workers. Exemplar workplace-related SARS-CoV-2 benchmarks are described and illustrated with empirical data. Methods: Benchmarks were collected over a 9-month period on a large workplace (N = 5500+). These ranged from quantitative indices associated with RT-qPCR targeted testing and random surveillance screening, surveillance for new variants of SARS-CoV-2, intensive contact tracing, case management, return to work procedures, to monitoring of antibody seropositive status. Results: Data and analyses substantiated effectiveness of interventions. This was evidenced in suppressed infection rates, rapid case identification and isolation, acceptance of the program by employees, documentation of presumptive immunity, and working relationships with senior management. Conclusions: These SARS-CoV-2 exemplar benchmarks provided an evidence-base for practice and contributed strategically to organizational decisions.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2019

Direct Randomized Benchmarking for Multiqubit Devices

Benchmarking methods that can be adapted to multiqubit systems are essential for assessing the overall or “holistic” performance of nascent quantum processors. The current industry standard is Clifford randomized benchmarking (RB), which measures a single error rate that quantifies overall performance. But, scaling Clifford RB to many qubits is surprisingly hard. It has only been performed on one, two, and three qubits as of this writing. This reflects a fundamental inefficiency in Clifford RB: the n-qubit Clifford gates at its core have to be compiled into large circuits over the one- and two-qubit gates native to a device. As n grows, the quality of these Clifford gates quickly degrades, making Clifford RB impractical at relatively low n. In this Letter, we propose a direct RB protocol that mostly avoids compiling. Instead, it uses random circuits over the native gates in a device, which are seeded by an initial layer of Clifford-like randomization. We demonstrate this protocol experimentally on two to five qubits using the publicly available ibmqx5. We believe this to be the greatest number of qubits holistically benchmarked, and this was achieved on a freely available device without any special tuning up. Our protocol retains the simplicity and convenient properties of Clifford RB: it estimates an error rate from an exponential decay. But, it can be extended to processors with more qubits—we present simulations on 10+ qubits—and it reports a more directly informative and flexible error rate than the one reported by Clifford RB. Here, we show how to use this flexibility to measure separate error rates for distinct sets of gates, and we use this method to estimate the average error rate of a set of cnot gates.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2024

Quapprox: A Framework for Benchmarking the Approximability of Variational Quantum Circuit

Most of the existing quantum neural network models, such as variational quantum circuits (VQCs), are limited in their ability to explore the non-linear relationships in input data. This gradually becomes the main obstacle for it to tackle realistic applications, such as natural language processing, medical image processing, and wireless communications. Recently, there have emerged research efforts that enable VQCs to perform non-linear operations. However, it is still unclear on the approximability of a given VQC (i.e., the order of non-linearity that can be handled by a specified design). In response to this issue, we developed an automated tool designed to benchmark the approximation of a given VQC. The proposed tool will generate a set of synthetic datasets with different orders of non-linearity and train the given VQC on these datasets to estimate their approximability. Our experiments benchmark VQCs with different designs, where we know their theoretic approximability. We then show that the proposed tool can precisely estimate the approximability, which is consistent with the theoretic value, indicating that the proposed tool can be used for benchmarking the approximability of a given quantum circuit for learning tasks.

artificial intelligence↗
DOE OSTI2024

Benchmarking Variables for Checkpointing in HPC Applications

Checkpoint/Restart (C/R) is a widely used fault tolerance mechanism in converged systems of cloud, edge, and HPC. However, users often rely on their experience to determine which variables to checkpoint, as there is currently no benchmark that can provide a reference. This can result in checkpointing redundant or even incorrect variables. To address this issue, we propose a benchmark suite that includes critical variables for checkpointing, which have been manually identified, and a method for identifying those critical variables, with 20 representative HPC applications. Our method involves analyzing data dependency between variables to identify critical variables analytically. We verify the identified variables' correctness with a widely used C/R library FTI by an ablation study. With our benchmark suite and data dependency analysis, HPC practitioners now have a reference for identifying checkpointing variables and better knowledge of what kind of variables to checkpoint.

Fu, Xiang↗
DOE OSTI2022

Climbing the Summit and Pushing the Frontier of Mixed Precision Benchmarks at Extreme Scale

The rise of machine learning (ML) applications and their use of mixed precision to perform interesting science are driving forces behind AI for science on HPC. The convergence of ML and HPC with mixed precision offers the possibility of transformational changes in computational science. The HPL-AI benchmark is designed to measure the performance of mixed precision arithmetic as opposed to the HPL benchmark which measures double precision performance. Pushing the limits of systems at extreme scale is nontrivial -little public literature explores optimization of mixed precision computations at this scale. In this work, we demonstrate how to scale up the HPL-AI benchmark on the pre-exascale Summit and exascale Frontier systems at the Oak Ridge Leadership Computing Facility (OLCF) with a cross-platform design. We present the implementation, performance results, and a guideline of optimization strategies employed for delivering portable performance on both AMD and NVIDIA GPUs at extreme scale.

Lu, Hao↗
DOE OSTI2025

LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators

Large Language Models (LLMs) have propelled groundbreaking advancements across several domains and are commonly used for text generation applications. However, the computational demands of these complex models pose significant challenges, requiring efficient hardware acceleration. Benchmarking the performance of LLMs across diverse hardware platforms is crucial to understanding their scalability and throughput characteristics. We introduce LLM-Inference-Bench, a comprehensive benchmarking suite to evaluate the hardware inference performance of LLMs. We thoroughly analyze diverse hardware platforms, including GPUs from Nvidia and AMD and specialized AI accelerators, Intel Habana and SambaNova. Our evaluation includes several LLM inference frameworks and models from LLaMA, Mistral, and Qwen families with 7B and 70B parameters. Our benchmarking results reveal the strengths and limitations of various models, hardware platforms, and inference frameworks. We provide an interactive dashboard to help identify configurations for optimal performance for a given hardware platform.

Chitty-Venkata, Krishna Teja↗
DOE OSTI2025

Benchmarking large language models for materials synthesis: The case of atomic layer deposition

In this work, we introduce an open-ended question benchmark, ALDbench, to evaluate the performance of large language models (LLMs) in materials synthesis, and, in particular, in the field of atomic layer deposition, a thin film growth technique used in energy applications and microelectronics. Our benchmark comprises questions with a level of difficulty ranging from the graduate level to domain expert current with the state of the art in the field. Human experts reviewed the questions along the criteria of difficulty and specificity, and the model responses along four different criteria: overall quality, specificity, relevance, and accuracy. We ran this benchmark on an instance of OpenAI’s GPT-4o. The responses from the model received a composite quality score of 3.7 on a 1–5 scale, consistent with a passing grade. However, 36% of the questions received at least one below average score. An in-depth analysis of the responses identified at least five instances of suspected hallucination. Finally, we observed statistically significant correlations between the difficulty of the question and the quality of the response, the difficulty of the question and the relevance of the response, the specificity of the question, and the accuracy of the response as graded by the human experts. Furthermore, this emphasizes the need to evaluate LLMs across multiple criteria beyond difficulty or accuracy.

Artificial intelligence↗
DOE OSTI2022

NISQ Benchmarking

Test suite of quantum algorithms for Noisy Intermediate Scale Quantum (NISQ) computers. The test suite includes benchmark-style code for quantum volume circuits (QV), fairness sampling circuits, quantum telecloning circuits, and other NISQ benchmark style algorithms on small problems (i.e., up to 100 qubits), such as Variational Quantum Eigensolver (VQE), Hamiltonian Simulation, and Grover unstructured search example circuits. These benchmark-style applications are implemented in quantum software packages, mostly IBM's QISKIT, but may include vendor-specific frameworks, such as PyQuil (for Rigetti) or Q\# for Microsoft, or CirQ (for Google) as the test suite grows with the vendor sample. The test suite also includes numerical simulation code for Quantum Alternating Operator Ansatz (QAOA) algorithms, VQE, Hamiltonian Simulation and search examples. Numerical simulation code simulates quantum computers on classical computers, which is only possible for small problem instances; the implementation framework of choice is typically within Python, using the numpy/scipy libraries as well as extensions to the Julia language.

Pelofske, Elijah↗
DOE OSTI2023

High-Performance GMRES Mixed-Precision (HPG-MxP) Benchmark

SAND2024-08539O The High Performance GMRES Mixed-Precision (HPG-MxP) is a benchmark for ranking high-performance supercomputers, allowing use of mixed-precision. Similar to HPCG benchmark, it is designed to profile the computers' capabilities to perform the computational and communication tasks that are commonly found in important classes of real-world applications. At the same time, like HPL-MxP benchmark, it allows the use of mixed-precision arithmetic, while ensuring the double-precision accuracy of the computed solution. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

SciDAC↗
DOE OSTI2025

Binder-benchmarking

SAND2025-07593O Binder-benchmarking evaluates the speed and memory impacts of C++, Python, and Matlab code binders. As a repository, it provides a way to locally run computation-based and memory-based benchmark suites on pybind11 and nanobind-based code in a Docker image. The software runs simple-speed and memory benchmarks on primitive navigation and integration exemplar algorithms. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Walker II, Michael [Sandia National Lab. (SNL-CA),↗
DOE OSTI2019

Benchmarking operational efficiency in waste collection: Discussion of current approaches and possible alternatives

Efficiency assessment and benchmarking are crucial for managing any organization. However, especially from a regulatory perspective, such efficiency assessment and benchmarking must be unbiased from context-specific issues and should provide an absolute rating, rather than a relative one. The current work reviews the approaches used for performance assessment and benchmarking waste collection services, revealing that the majority are biased and are not absolute, and proposes two alternative context-unbiased and absolute performance indicators, the collection capacity use (CCU) and the segregated waste collection efficiency (SWE). The proposed indicators were calculated for 246 utilities operating in Portugal. The utilities were then ranked accordingly, and their position was compared with the position attained using the equivalent performance indicators in the system currently in use by the Portuguese service regulator. The results reveal ranking differences of over 50 positions and illustrate how misleading the results from context-biased and relative metrics can be.

Sousa, V.↗
DOE OSTI2022

Benchmark Analysis

Tremendous progress has been achieved in the development of land models and their inclusion in Earth system models (ESMs). However, we still have very limited knowledge on the performance skills of these land models. This chapter introduces benchmark analysis, which is a procedure to measure performance of models against a set of defined standards. The benchmark analysis includes: (1) defining targeted aspects of model performance to be evaluated; (2) testing model performance in comparison with a set of benchmarks; (3) measuring model performance skill through quantitative metrics; and (4) evaluating model performance and offering suggestions for future model improvement.

Luo, Yiqi↗
DOE OSTI2021

Development of a Pulsed Slowing-Down-Time Benchmark of Neutron Thermalization in Graphite

Graphite is a classic neutronic material that has been used as both a reactor reflector and moderator in various nuclear reactor systems. The ability to accurately predict the slowing down and thermalization of neutrons in graphite can have significant implications on the safety and operation of such reactor systems. In reactors, the neutron thermalization process is quantified using the thermal scattering law (TSL) and related cross sections for a given moderator. An ideal approach to assess the validity of TSL data is using benchmark measurement based on the pulsed Slowing-Down-Time technique and its comparison with the appropriate graphite nuclear library. In this work, experimental measurement and computational Monte Carlo simulations were performed to benchmark the slowing down characteristics and thermalization of neutrons in nuclear (reactor-grade) graphite. Given the density of graphite, various graphite libraries were selected for the benchmark analysis.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

bmdrc: Python package for quantifying phenotypes from chemical exposures with benchmark dose modeling

Though chemical exposures are known to potentially have negative impacts on health, including contributing to chronic diseases such as cancer, the quantitative contribution of risk is not fully understood for every chemical. A commonly used approach to quantify levels of risk is to measure the proportion of organisms (such as a total number of zebrafish on a plate or mice in a cage) with abnormal behavioral responses or morphology at increasing concentrations of chemical exposure. A particular challenge with processing the proportional data from these assays is the appropriate estimation of chemical concentration levels that result in malformations or acute toxicity, as these values typically vary between experimental measurements. The recommended approach by the Environmental Protection Agency (EPA) is to fit benchmark dose curves with specific filters and model fitting steps, which are crucial to properly processing the proportional data. Several tools exist for the fitting of benchmark dose response curves, but none are standalone Python libraries built to process both morphological and behavioral data as proportions with all the EPA recommended filters, filter parameters, models, and model parameters. Thus, here we present the benchmark dose response curve (bmdrc) Python library, which was built to closely follow these EPA guidelines with helpful visualizations of filters and fitted model curves, and reports for reproducibility purposes. bmdrc is open-source and has demonstrated utility as a support package to an existing web portal for information on chemicals (https://srp.pnnl.gov). Our package will support any toxicology analysis where the response is a proportional value at increasing levels of a concentration of a chemical or chemical mixture.

Superfund↗
DOE OSTI2018

Integrated FHR technology development: Tritium management, materials testing, salt chemistry control, thermal hydraulics and neutronics, associated benchmarking and commercial basis

This report summarizes the results of a 3-year Integrated Research Project (IRP) sponsored by the U.S. Department of Energy to address major technical challenges in the development of the Fluoride-salt-cooled High-temperature Reactor (FHR). The IRP universities included the Massachusetts Institute of Technology (MIT), the University of California at Berkeley (UCB), the University of Wisconsin (UW), and the University of New Mexico (UNM). The FHR is a new reactor concept less than 20 years old that combines (1) a clean fluoride salt coolant, (2) the graphite-matrix coated-particle fuel originally developed for High- Temperature Gas-Cooled Reactors and (3) passive decay heat removal systems from sodium fast reactors. The base-line design is a pebble-bed FHR that delivers heat between 600 to 700°C with a 7 Li 2 BeF 4 (flibe) salt coolant to the power cycle or industrial users. The Executive Summary describes what was accomplished and summarizes major conclusions. The main report contains stand-alone sections in each major area of work. Highlights are summarized below. Tritium. The coolant salt produces tritium necessitating understanding and control of tritium. Experimental work included measuring hydrogen uptake on carbon, tritium uptake during irradiation of graphite in 700°C salt and the permeability of tritium through metals with and without coatings. An FHR systems model was developed to predict tritium behavior and enable design of tritium control systems. The leading candidate for tritium removal from the flibe salt is carbon bed outside the reactor core. Thermal hydraulics and Neutronics. A 10-meter high FHR integral effects test facility, using Dowtherm A®, has been built and used to simulate the integral response of a representative FHR to improve understanding of FHR thermal hydraulics and validate thermal hydraulics codes used for the FHR. Neutronic analysis included code-to-code benchmarking, multiphysics modeling, uncertainty analysis for nuclear data uncertainties and shielding analysis. Twisted-tube heat exchangers were examined that may provide significant cost reductions because of their improved heat-transfer performance with liquid salts. The impacts on safety and design caused by uncertainties in physical properties of the salt were assessed with recommendations on where added physical property measurements are needed. Work began on understanding radiative heat transfer that becomes important in salt systems at higher temperatures. Materials. Static corrosion tests were conducted on graphite and materials in 700°C salt -- both in the UW laboratory and in the MIT reactor. In addition to static corrosion testing, a circulating salt corrosion loop was built and is operating. Methods to measure salt properties including redox were developed. The presence of carbon increases corrosion. With good redox control, the evidence indicates stainless steel can be used as a material of construction. Benchmarking. An additional series of workshops where held to enable benchmarking and comparison of results between different investigators. Commercialization. The FHR delivers higher average-temperature (~650°C) heat to the power cycle or industry than other reactors. This enables power cycles with lower-cost heat storage for variable electricity to the grid while the reactor operates at base-load -- providing a major competitive advantage by maximizing revenue from electricity sales. In October, 2016, a new startup company named Kairos Power was formed, and opened its first office in Jack London Square, Oakland, California, in January 2017. Kairos Power exists largely due to the work that was accomplished by this IRP and the previous DOE-funded FHR IRP that performed research on FHR technology from 2012 to 2015. In September, 2018, as this IRP is concluding, Kairos Power was staffed at 60 full time employees, working to develop technology derived from this IRP. Many of these employees worked with the IRP before joining Kairos Power 1 (https://kairospower.com). Other students and researchers from this IRP have joined other startup companies, including the Terrapower Molten Chloride Fast Reactor and the Terrestrial Energy development efforts.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2019

Nuclear Data and Benchmarking Program: Nuclear Data Performance Assessment for Advanced Reactors

Over the last several Evaluated Nuclear Data File (ENDF)/B releases, many updates have been introduced to the nuclear data libraries that can significantly change computational results. These changes can be particularly important for advanced reactor concepts, which are not as thoroughly investigated as those used in light water reactor (LWR) systems. Therefore, a performance assessment of the ENDF/B libraries using the SCALE 6.2.3 code package was conducted under the auspices of the U.S., Department of Energy Office of Nuclear Energy (DOE-NE). The following systems relevant to the advanced reactor community were chosen as subject for investigation in this report: sodium cooled fast reactor systems, graphite moderated high temperature gas-cooled reactors, and several molten salt reactor models. At first, the similarity of the integral performance of the ENDF/B-VII.1 and ENDF/B-VIII.0 nuclear data libraries caused by compensating errors in important differential data was demonstrated. In calculations for a shipping container of high-assay low enriched uranium, the data of individual isotopes were systematically swapped between the two ENDF/B libraries. An eigenvalue difference of up to 450 pcm was found due to the use of 235 U and 238 U from one library and 1 H and 16 O from the other library. This clearly demonstrates a cross-correlation between reaction data sets of different isotopes within a library that should be reported in the evaluations. For sodium-cooled fast reactor (SFR) fuel assemblies, significant eigenvalue (k inf ) differences (200– 450 pcm) were found between calculations using the 2011 ENDF/B-VII.1 data and 2018 ENDF/B-VIII.0 data. These differences were mainly caused by updates of 238 U and 239 Pu neutron cross sections. Results from the multigroup (MG) calculations further revealed that the group structure of the applied MG library strongly influences the MG bias due to the importance of an appropriate energy resolution of the resonances in higher energy ranges. The application of previously used MG libraries requires new verification for advanced reactor simulation in comparisons with reference continuous-energy calculations. Nuclear data uncertainty analyses of these SFR systems resulted in eigenvalue uncertainties between 1,400 and 1,800 pcm, which are three to four times higher than corresponding uncertainties in light water reactor (LWR) systems. The main contributor to this uncertainty was found to be inelastic scattering on 238 U, which shows an uncertainty of up to 50% in the fast energy range. Other important contributors are the scattering reactions of 56 Fe and 23 Na which have so far not appeared in LWR analysis. Since the largest contribution to the eigenvalue uncertainty of SFR systems is coming from scattering reactions, it is expected that uncertainties in the angular scattering distributions also have a significant impact. Those uncertainties are, however, not available for the majority of nuclides, and the capabilities to determine sensitivities to this data are currently underdeveloped. For two graphite moderated high temperature gas-cooled reactor (HTGR) benchmarks, only the ENDF/BVII. 1 calculations resulted in consistent eigenvalues (k eff ) with the corresponding experimental measurements. Eigenvalue differences of about 1,000 pcm were found between the 2006 ENDF/B-VII.0 data and the 2011 ENDF/B-VII.1 data, primarily due to an updated carbon capture cross section in the thermal energy range in the later ENDF/B release. The ENDF/B-VIII.0 eigenvalue was larger than the ENDF/B-VII.1 eigenvalue by about 300 pcm mainly due to updates in multiple 235 U neutron cross sections, with offsetting updates in the 238 U cross sections. With ENDF/B-VIII.0, graphite can be modeled as perfect crystal or with two different porosities. The choice of the graphite evaluation can have a significant influence of the eigenvalue. For a HTGR benchmark, maximum eigenvalue differences as high as 650 pcm due to different porosities were observed. More detailed studies of the impact of the porosity are necessary while considering that the graphite porosity can vary between the used materials and changes as a function of neutron fluence. xii The uncertainty of the HTGR eigenvalues due to nuclear data uncertainties was found between 500 and 600 pcm, with the top contributor being the neutron multiplicity of 235 U. A gap in the form of missing uncertainties in graphite thermal scattering data, that might have significant impact on HTGR reactors, was identified. This gap is being investigated by the DOE-NE Nuclear Data and Benchmarking Program, as documented in a separate report The fast spectrum molten salt reactor calculations revealed similar comparisons as for the SFR assemblies. The eigenvalue uncertainties were significantly influenced by inelastic scattering on 238 U, while the impact of angular scattering distributions is unknown. For the graphite and zirconium hydride moderated system, eigenvalue uncertainties of up to 700 pcm were observed, requiring in depth analyses of the cross sections and corresponding uncertainties of the included materials. In general, it was observed that the flux spectra of the various molten salt reactor systems show significant differences, even between a fresh state and a depleted state. The choice of the energy group structure of MG calculations is therefore highly relevant; the applicability of previously used MG libraries needs to be verified. Due the unavailability of pin power measurements of advanced reactor systems, pin power calculations with both ENDF/B-VII.1 and VIII.0 data were performed for a light water reactor and compared to corresponding measurements. Both calculations show good agreement with the measurements. A comparison between the ENDF/B-VII.1 and VIII.0 results revealed small differences, mostly in the range of about 0.2%.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2019

Critical Experiment Benchmark Results using MCNP6.2 Unstructured Mesh

The purpose of this report is to document the results of five critical experiment benchmarks modeled using the unstructured mesh (UM) geometry capability in MCNP®1 code, version 6.2. LANL currently has a vast critical experiment benchmark library with over 1100 cases, modeled using constructive solid geometry (CSG) and packaged with Whisper-1.1. Validation efforts for the purposes of nuclear criticality safety typically focus on benchmark libraries built using CSG models. Results of UM geometry models are compared with CSG results and the critical experiment k-effective from the ICSBEP handbook. All computed k-effective results in this study are well within 1% of the experimental result.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗