Search NASA⌕ Search

SEARCH · Search NASA

Results for “BENCHMARK”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators

Large Language Models (LLMs) have propelled groundbreaking advancements across several domains and are commonly used for text generation applications. However, the computational demands of these complex models pose significant challenges, requiring efficient hardware acceleration. Benchmarking the performance of LLMs across diverse hardware platforms is crucial to understanding their scalability and throughput characteristics. We introduce LLM-Inference-Bench, a comprehensive benchmarking suite to evaluate the hardware inference performance of LLMs. We thoroughly analyze diverse hardware platforms, including GPUs from Nvidia and AMD and specialized AI accelerators, Intel Habana and SambaNova. Our evaluation includes several LLM inference frameworks and models from LLaMA, Mistral, and Qwen families with 7B and 70B parameters. Our benchmarking results reveal the strengths and limitations of various models, hardware platforms, and inference frameworks. We provide an interactive dashboard to help identify configurations for optimal performance for a given hardware platform.

Chitty-Venkata, Krishna Teja↗
DOE OSTI2025

Benchmarking large language models for materials synthesis: The case of atomic layer deposition

In this work, we introduce an open-ended question benchmark, ALDbench, to evaluate the performance of large language models (LLMs) in materials synthesis, and, in particular, in the field of atomic layer deposition, a thin film growth technique used in energy applications and microelectronics. Our benchmark comprises questions with a level of difficulty ranging from the graduate level to domain expert current with the state of the art in the field. Human experts reviewed the questions along the criteria of difficulty and specificity, and the model responses along four different criteria: overall quality, specificity, relevance, and accuracy. We ran this benchmark on an instance of OpenAI’s GPT-4o. The responses from the model received a composite quality score of 3.7 on a 1–5 scale, consistent with a passing grade. However, 36% of the questions received at least one below average score. An in-depth analysis of the responses identified at least five instances of suspected hallucination. Finally, we observed statistically significant correlations between the difficulty of the question and the quality of the response, the difficulty of the question and the relevance of the response, the specificity of the question, and the accuracy of the response as graded by the human experts. Furthermore, this emphasizes the need to evaluate LLMs across multiple criteria beyond difficulty or accuracy.

Artificial intelligence↗
DOE OSTI2022

NISQ Benchmarking

Test suite of quantum algorithms for Noisy Intermediate Scale Quantum (NISQ) computers. The test suite includes benchmark-style code for quantum volume circuits (QV), fairness sampling circuits, quantum telecloning circuits, and other NISQ benchmark style algorithms on small problems (i.e., up to 100 qubits), such as Variational Quantum Eigensolver (VQE), Hamiltonian Simulation, and Grover unstructured search example circuits. These benchmark-style applications are implemented in quantum software packages, mostly IBM's QISKIT, but may include vendor-specific frameworks, such as PyQuil (for Rigetti) or Q\# for Microsoft, or CirQ (for Google) as the test suite grows with the vendor sample. The test suite also includes numerical simulation code for Quantum Alternating Operator Ansatz (QAOA) algorithms, VQE, Hamiltonian Simulation and search examples. Numerical simulation code simulates quantum computers on classical computers, which is only possible for small problem instances; the implementation framework of choice is typically within Python, using the numpy/scipy libraries as well as extensions to the Julia language.

Pelofske, Elijah↗
DOE OSTI2023

High-Performance GMRES Mixed-Precision (HPG-MxP) Benchmark

SAND2024-08539O The High Performance GMRES Mixed-Precision (HPG-MxP) is a benchmark for ranking high-performance supercomputers, allowing use of mixed-precision. Similar to HPCG benchmark, it is designed to profile the computers' capabilities to perform the computational and communication tasks that are commonly found in important classes of real-world applications. At the same time, like HPL-MxP benchmark, it allows the use of mixed-precision arithmetic, while ensuring the double-precision accuracy of the computed solution. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

SciDAC↗
DOE OSTI2025

Binder-benchmarking

SAND2025-07593O Binder-benchmarking evaluates the speed and memory impacts of C++, Python, and Matlab code binders. As a repository, it provides a way to locally run computation-based and memory-based benchmark suites on pybind11 and nanobind-based code in a Docker image. The software runs simple-speed and memory benchmarks on primitive navigation and integration exemplar algorithms. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Walker II, Michael [Sandia National Lab. (SNL-CA),↗
DOE OSTI2022

Benchmark Analysis

Tremendous progress has been achieved in the development of land models and their inclusion in Earth system models (ESMs). However, we still have very limited knowledge on the performance skills of these land models. This chapter introduces benchmark analysis, which is a procedure to measure performance of models against a set of defined standards. The benchmark analysis includes: (1) defining targeted aspects of model performance to be evaluated; (2) testing model performance in comparison with a set of benchmarks; (3) measuring model performance skill through quantitative metrics; and (4) evaluating model performance and offering suggestions for future model improvement.

Luo, Yiqi↗
DOE OSTI2021

Development of a Pulsed Slowing-Down-Time Benchmark of Neutron Thermalization in Graphite

Graphite is a classic neutronic material that has been used as both a reactor reflector and moderator in various nuclear reactor systems. The ability to accurately predict the slowing down and thermalization of neutrons in graphite can have significant implications on the safety and operation of such reactor systems. In reactors, the neutron thermalization process is quantified using the thermal scattering law (TSL) and related cross sections for a given moderator. An ideal approach to assess the validity of TSL data is using benchmark measurement based on the pulsed Slowing-Down-Time technique and its comparison with the appropriate graphite nuclear library. In this work, experimental measurement and computational Monte Carlo simulations were performed to benchmark the slowing down characteristics and thermalization of neutrons in nuclear (reactor-grade) graphite. Given the density of graphite, various graphite libraries were selected for the benchmark analysis.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

bmdrc: Python package for quantifying phenotypes from chemical exposures with benchmark dose modeling

Though chemical exposures are known to potentially have negative impacts on health, including contributing to chronic diseases such as cancer, the quantitative contribution of risk is not fully understood for every chemical. A commonly used approach to quantify levels of risk is to measure the proportion of organisms (such as a total number of zebrafish on a plate or mice in a cage) with abnormal behavioral responses or morphology at increasing concentrations of chemical exposure. A particular challenge with processing the proportional data from these assays is the appropriate estimation of chemical concentration levels that result in malformations or acute toxicity, as these values typically vary between experimental measurements. The recommended approach by the Environmental Protection Agency (EPA) is to fit benchmark dose curves with specific filters and model fitting steps, which are crucial to properly processing the proportional data. Several tools exist for the fitting of benchmark dose response curves, but none are standalone Python libraries built to process both morphological and behavioral data as proportions with all the EPA recommended filters, filter parameters, models, and model parameters. Thus, here we present the benchmark dose response curve (bmdrc) Python library, which was built to closely follow these EPA guidelines with helpful visualizations of filters and fitted model curves, and reports for reproducibility purposes. bmdrc is open-source and has demonstrated utility as a support package to an existing web portal for information on chemicals (https://srp.pnnl.gov). Our package will support any toxicology analysis where the response is a proportional value at increasing levels of a concentration of a chemical or chemical mixture.

Superfund↗
DOE OSTI2020

Benchmark problems for the Me soscale M ultiphysics Ph ase F i eld S imulator (MEMPHIS)

This report details the current benchmark results to verify, validate and demonstrate the capabilities of the in-house multi-physics phase-field modeling framework Me soscale M ultiphysics Ph ase F i eld S imulator (MEMPHIS) developed at the Center for Integrated Nanotechnologies (CINT). MEMPHIS is a general phase-field capability to model various nanoscience and materials science phenomena related to microstructure evolution. MEMPHIS has been benchmarked against a suite of reported 'classical' phase-field benchmark problems to verify and validate the correctness, accuracy and precision of the models and numerical methods currently implemented into the code.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2020

Comparison of Modeled to Measured Spectra using MCNP and GADRAS to Benchmark and Contrast Modeling Limitations

The desire to improve nuclear material detection through portal monitors and other low resolution detectors has led to interest in benchmarking the performance of radiation transport codes. These codes can be used to generate a variety and quantity of spectra that may be cost prohibitive to measure directly. Particular characteristics of typical detection scenarios were isolated to compare the performance of radiation transport codes to laboratory experiments. These benchmark experiments were performed to validate simulations in a number of key areas. These experiments included high areal density configurations in both one- and three-dimensional configurations. In addition, off-angle measurements were conducted to investigate the impact of detector response assumptions on complex geometries, for example where an unknown source is not physically collocated with an apparent hot spot. Furthermore, by examining both simple shielding and backscatter configurations, isolation of the scatter emanating from the object and environmental background scatter contributions to the spectra could be elucidated. A series of experiments were performed in which the order of shielding layers and the thickness were evaluated to examine the ability of the simulations to address these variables. Experiments with increasing thickness of polyethylene around a neutron source also allows for examinations of the ability of the transport simulations to properly model n,γ reactions. Finally, depleted uranium (DU) measurements have more complicated spectra than the cobalt-60 and consequently allow for investigations into the ability of transport simulations to model self-shielding. In each case, the results of the experiments were compared to MCNP simulations to judge the performance of the code, as was done previously with GADRAS 18.7.9 simulations [7]. All of the benchmark measurements were taken with a liquid nitrogen cooled 140% high purity germanium (HPGe) detector with a bismuth collimator that had the front tin filter removed. The measurement location and detector configuration were subsequently used for all experimental configurations. The remainder of this report details the experimental measurements, Section 2 and the MCNP model, Section 3. Comparisons between the experimental results and MCNP-generated spectra are performed, Section 4. Section 5 compares the results obtained with MCNP with the previously reported results obtained with GADRAS [2]. Finally, Section 6 presents the conclusions.

61 RADIATION PROTECTION AND DOSIMETRY↗
DOE OSTI2020

A Metallic Fuel Performance Benchmark Problem Based on the IFR-1 Experiment

Metallic nuclear fuels are an active area of research and development for use in advanced reactors. Robust and accurate metallic fuel performance models are necessary for the design, safety analysis, and licensing of such reactors. However, metallic fuel performance models are not as mature as ceramic fuel performance models and require additional development. In order to better gauge the accuracy of existing models, identify high-priority models for development, and quantify the improvements made by future model development, ORNL has developed a benchmark problem based on the IFR-1 experiment. This work gathers all relevant information on the IFR-1 experiment and uses it to develop the benchmark problem. It simulates the problem using the fuel performance code BISON and compare the results to post-irradiation examination data from the experiment. Then a sensitivity analysis is performed on the BISON model to determine the benchmark problem’s sensitivity to uncertainty in the input parameters. The results suggest that BISON’s mechanical models require additional development. Plastic deformation is under-predicted in the cladding and axial swelling is over-predicted in the fuel. These problems may be related. Additionally, there is a bias in the temperature which may be a result of uncertainty in the input parameters rather than an issue with the fuel performance models.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2020

GoldSim E-Area Low-Level Waste Lacility Vadose Zone Model Benchmarking

This report documents the development and benchmarking of the E-Area Low-Level Waste Facility (ELLWF) GoldSim Engineered and Slit Trench (ET and ST) vadose zone models. Subsequent activities (beyond the scope of this effort) will couple the GoldSim-based vadose zone models to the GoldSim aquifer model, with the intent of performing stochastic analyses that couple all pertinent fate and transport processes from the ground surface up to the 100-m Point of Assessment (POA). The benchmarking presented within this report focuses on the Slit Trench 06 (ST06) bydrostratigraphic location that resides within the center set of slit trenches. These GoldSim-based vadose zone models are generic models capable of being benchmarked to any of the other trench locations within E-Area.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2020

Benchmark problems for the Mesoscale Multiphysics Phase Field Simulator (MEMPHIS)

This report details the current benchmark results to verify, validate and demonstrate the capabilities of the in-house multi-physics phase-field modeling framework Mesoscale Multiphysics Phase Field Simulator (MEMPHIS) developed at the Center for Integrated Nanotechnologies (CINT). MEMPHIS is a general phase-field capability to model various nanoscience and materials science phenomena related to microstructure evolution. MEMPHIS has been benchmarked against a suite of reported classical phase-field benchmark problems to verify and validate the correctness, accuracy and precision of the models and numerical methods currently implemented into the code.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2021

Wind Plant Performance Prediction Benchmark Phase 1 (Technical Report)

Financial risk resulting from the uncertainty associated with developing, owning, and operating wind power plants remains a barrier to reducing the levelized cost of energy (LCOE). On average, modern wind power plants in the U.S. underperform their expected annual energy output by 3.5-4.5% , with many underperforming by over 10%. To compensate for this uncertainty, investors require a larger return on investment (ROI) and apply "knock-down" factors that mask much of the underlying sources of uncertainty. Wind energy projects thus have reduced access to low-cost capital. Furthermore, operating wind plants often take a simple approach to estimating operations & maintenance (O&M) costs (e.g. straight-line estimates based on similar plants), which can eat into profits. To overcome these issues, the wind industry must improve the models they use for estimating wind plant performance and operations. An industry consortium (IC) requested that the National Renewable Energy Laboratory (NREL) lead a Department of Energy (DOE) working group to benchmark the accuracy of wind power plant energy predictions against real operational data. The IC was also motivated by DOE and NREL's potential to characterize systematic energy underperformance, identify sources of uncertainty, and explore root causes. The Wind Plant Performance Prediction (WP3) project was created out of this request, and this report represents the successful completion of Phase 1 of the WP3 project. During the project, wind plant owners provided both pre-construction and operational data to NREL. The pre-construction data was provided to wind resource assessment (WRA) consultants so they could conduct energy yield assessments (EYA). NREL took all of the completed EYAs, along with the operational data, and conducted an operational assessment to benchmark the EYA results against actual operational data. Given the large amounts of sensitive data required for this effort, as well as historical opposition to sharing data within industry, successful completion of Phase 1 represents an unprecedented milestone for industry data sharing. To improve the accuracy and confidence of pre-construction EYAs, wind plant owners and investors need better, more certain, energy yield predictions. The WP3 Benchmark Project is an industry-driven response to this reality. For the first time, industry has taken the important step of working together at scale, sharing valuable operational data with DOE and NREL in order to investigate the sources of bias and uncertainty in these energy estimates. This IC provides wind plant preconstruction and operational data to NREL in an organized and documented fashion and provides guidance and feedback as needed. The IC also provides introspection of the design of experiment, key metrics of success, data challenges, analysis best practices, and quality of results.

17 WIND ENERGY↗
DOE OSTI2021

U.S. Solar Photovoltaic System and Energy Storage Cost Benchmarks: Q1 2021

Based on our bottom-up modeling, the Q1 2021 PV and energy storage cost benchmarks are: $\$2.65$ per watt DC (WDC) (or $\$3.05$/WAC) for residential PV systems, 1.56/WDC (or $\$1.79$/WAC) for commercial rooftop PV systems, $\$1.64$/WDC (or $\$1.88$/WAC) for commercial ground-mount PV systems, $\$0.83$/WDC (or $\$1.13$/WAC) for fixed-tilt utility-scale PV systems, $\$0.89$/WDC (or $\$1.20$/WAC) for one-axis-tracking utility-scale PV systems, $\$30,326$-$\$33,618$ for a 7.15-kWDC residential PV system with 5 kW/12.5 kWh nameplate of storage, $\$2.04$ - $\$2.10$ million for a 1-MWDC commercial ground-mount PV system colocated with 600 kW/2.4 MWhusable of storage, $\$166$ - $\$167$ million for a 100-MWDC one-axis tracker PV system colocated with 60 MW/240 MWhusable of storage. Between 2020 and 2021, there were 3.3% ($\$0.0$9/W), 10.7% ($\$0.19$/W), and 12.3% ($\$0.13$/W) reductions (in 2020 USD) in the residential, commercial rooftop, and utility-scale (one-axis) PV system cost benchmarks respectively. Balance of system (BOS) costs have either increased or remained flat across sectors, year-on-year, unlike in previous benchmark reports, which generally have reported declining BOS costs. The increase in BOS cost has been offset by a 17% reduction in module cost. Overall, modeled PV installed costs across the three sectors have declined compared to our Q1 2020 system costs.

14 SOLAR ENERGY↗
DOE OSTI2021

Initial Benchmarks of UV LEDs and Comparisons with White LEDs

The primary goal of this report is to benchmark the initial level of performance of a selection of commercial UV LEDs across all three bands (i.e., UV-A, UV-B, UV-C). To provide the initial performance benchmarks, a test matrix containing 13 different UV LED products was created in association with the LED Systems Reliability Consortium (LSRC). The products in this test matrix were all commercially available as of June 2021, and at least 22 samples of each product were tested. In addition, two common, commercial white LEDs were tested to provide a benchmark against blue-pumped white LEDs. Testing of the samples included electrical performance testing (e.g., current-voltage measurements) and photometric testing in a calibrated integrating sphere capable of measuring devices in the UV-A, UV-B, and UV-C bands. The electrical testing provided insights into the performance of the semiconductor layers in the LEDs, allowing parameters such as the threshold voltage (V th ) and serial resistance (R serial ) of each sample to be determined. The photometric testing provided insights into emission wavelengths, peak shapes, and radiant efficiencies for each sample. Combined, the information from these tests permits the overall device efficiencies to be compared and provides insights into the electrical and optical performance of the technology.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2022

Benchmark Specification for FFTF LOFWOS Test #13

The Fast Flux Test Facility (FFTF) at the Hanford site in Washington was designed by the Westinghouse Electric Corporation for the U.S. Department of Energy. FFTF was a 400 MW thermal, oxide-fueled, liquid sodium cooled test reactor, built to assist development and testing of advanced fuels and materials for fast breeder reactors. After reaching criticality in 1980, FFTF operated until 1992, providing the U.S. Department of Energy (DOE) with the means to test fuels, materials, and other components in a fast neutron flux environment. In July 1986, a series of unprotected transients (with the plant protection system intentionally disabled) were performed in FFTF as part of the passive safety demonstration program. Among these were thirteen loss of flow without scram (LOFWOS) tests. The goals of this program included confirming the liquid metal reactor safety margins, providing data for computer code validation, and demonstrating the inherent and passive safety benefits of specific design features. The test defined in this benchmark is LOFWOS Test #13, which was initiated at 50% power and 100% flow with the pump pony motors turned off. This benchmark specification is intended to support collaborative efforts within international partnerships on the validation of simulation tools and models in the area of Sodium-cooled Fast Reactor (SFR) safety. Validated tools and models are needed to evaluate SFR inherent safety characteristics and assess the impact of passive design features in response to accident initiators. Comparisons with experimental data and the results of safety analyses from other groups create unique opportunities to improve predictive capabilities of computational codes and methods for SFR modeling and simulation. The conditions of the LOFWOS test along with the feedback from FFTF’s limited free bow core restraint system and the novel passive reactivity control Gas-Expansion Modules (GEMs) pose a very challenging and uniquely valuable benchmark exercise.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2022

Benchmark Exercise for the Control Rod Swelling Evaluation

The VTR core has six reactivity control assemblies and three safety assemblies. The control assemblies or primary control rods are adjusted during the normal operation to balance the core reactivity and to control the reactor power. A typical control assembly radial layout is presented in Figure 1. The figure shows the swelled absorber (B 4 C) rod. Initially, helium gas fills the gap between the pin and the cladding before irradiation swelling takes place. For VTR, HT9 steel was selected as the cladding and duct material. The main neutron absorbing material used in the VTR is B 4 C. When residing in the core, the neutronics, thermophysical, and mechanical properties of the materials used in a control assembly will degrade due to accumulated neutron damage. Material degradation limits how long a control assembly can reside in the core. Many phenomena affect the control assembly lifetime, such as the loss of reactivity worth due to B 4 C depletion, the mechanical interaction of the absorber rod and the cladding due to B 4 C swelling, the helium gas buildup in the pin due to B-10 capture, etc. B 4 C swelling, which causes closure of the gap between the absorber rod and the cladding, is usually considered as the main limiting factor from past experience. An initial study was conducted at PNNL to evaluate the irradiation behavior of a VTR control assembly. The evaluation was performed using the CNRD2 code that was initially developed for the FFTF. The study also included an assessment of the VTR control assembly and focused on a 61-pin control assembly design, which is different from that used (37-pin design) in the core design study. The study conducted by PNNL was reviewed independently by ANL. A Python script referred to as the Control Assembly Evaluation Script (CAES) was developed for the independent review and additional assessment of 37-pin control assembly design. The script has focused on the assessment of the absorber rod swelling for its importance in determining the control assembly lifetime. CAES uses geometry, neutronics, materials data as input to predict the swelling of the absorber rod during its residence in the reactor core. The results from CAES showed some non-negligible differences against the PNNL results. Some of the differences can be attributed to the different interpretation of the control rod assembly dimensions. To resolve this issue, a benchmark exercise was proposed. The benchmark specification was developed by PNNL. The benchmark exercise was performed independently at PNNL and ANL using different codes/scripts (CRND2 and CAES). This memo documents the results calculated using the different codes. However, this report is limited to presenting the results obtained. Further investigation of the cause of the observed difference will be performed as part of future activities, pending continuation of the VTR program.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗