Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmark”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 361 records · Page 20

Benchmarking Variables for Checkpointing in HPC Applications

Checkpoint/Restart (C/R) is a widely used fault tolerance mechanism in converged systems of cloud, edge, and HPC. However, users often rely on their experience to determine which variables to checkpoint, as there is currently no benchmark that can provide a reference. This can result in checkpointing redundant or even incorrect variables. To address this issue, we propose a benchmark suite that includes critical variables for checkpointing, which have been manually identified, and a method for identifying those critical variables, with 20 representative HPC applications. Our method involves analyzing data dependency between variables to identify critical variables analytically. We verify the identified variables' correctness with a widely used C/R library FTI by an ablation study. With our benchmark suite and data dependency analysis, HPC practitioners now have a reference for identifying checkpointing variables and better knowledge of what kind of variables to checkpoint.

Fu, Xiang↗

Climbing the Summit and Pushing the Frontier of Mixed Precision Benchmarks at Extreme Scale

The rise of machine learning (ML) applications and their use of mixed precision to perform interesting science are driving forces behind AI for science on HPC. The convergence of ML and HPC with mixed precision offers the possibility of transformational changes in computational science. The HPL-AI benchmark is designed to measure the performance of mixed precision arithmetic as opposed to the HPL benchmark which measures double precision performance. Pushing the limits of systems at extreme scale is nontrivial -little public literature explores optimization of mixed precision computations at this scale. In this work, we demonstrate how to scale up the HPL-AI benchmark on the pre-exascale Summit and exascale Frontier systems at the Oak Ridge Leadership Computing Facility (OLCF) with a cross-platform design. We present the implementation, performance results, and a guideline of optimization strategies employed for delivering portable performance on both AMD and NVIDIA GPUs at extreme scale.

Lu, Hao↗

LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators

Large Language Models (LLMs) have propelled groundbreaking advancements across several domains and are commonly used for text generation applications. However, the computational demands of these complex models pose significant challenges, requiring efficient hardware acceleration. Benchmarking the performance of LLMs across diverse hardware platforms is crucial to understanding their scalability and throughput characteristics. We introduce LLM-Inference-Bench, a comprehensive benchmarking suite to evaluate the hardware inference performance of LLMs. We thoroughly analyze diverse hardware platforms, including GPUs from Nvidia and AMD and specialized AI accelerators, Intel Habana and SambaNova. Our evaluation includes several LLM inference frameworks and models from LLaMA, Mistral, and Qwen families with 7B and 70B parameters. Our benchmarking results reveal the strengths and limitations of various models, hardware platforms, and inference frameworks. We provide an interactive dashboard to help identify configurations for optimal performance for a given hardware platform.

Chitty-Venkata, Krishna Teja↗

Experimental validation benchmark data for CFD of transient convection from forced to natural with flow reversal on a vertical flat plate

Transient convection has been investigated experimentally for the purpose of providing Computational Fluid Dynamics (CFD) validation benchmark data. A specialized facility for validation benchmark experiments called the Rotatable Buoyancy Tunnel was used to acquire thermal and velocity measurements of flow over a smooth, vertical heated plate. The initial condition was forced convection downward with subsequent transition to mixed convection, ending with natural convection upward after a flow reversal. Data acquisition through the transient was repeated for ensemble-averaged results. With simple flow geometry, validation data were acquired at the benchmark level. All boundary conditions (BCs) were measured and their uncertainties quantified. Temperature profiles on all four walls and the inlet were measured, as well as as-built test section geometry. Inlet velocity profiles and turbulence levels were quantified using Particle Image Velocimetry. System Response Quantities (SRQs) were measured for comparison with CFD outputs and include velocity profiles, wall heat flux, and wall shear stress. Extra effort was invested in documenting and preserving the validation data. Details about the experimental facility, instrumentation, experimental procedure, materials, BCs, and SRQs are made available through this paper. As a result, the latter two are available for download and the other details are included in this work.

42 ENGINEERING↗

Benchmarking large language models for materials synthesis: The case of atomic layer deposition

In this work, we introduce an open-ended question benchmark, ALDbench, to evaluate the performance of large language models (LLMs) in materials synthesis, and, in particular, in the field of atomic layer deposition, a thin film growth technique used in energy applications and microelectronics. Our benchmark comprises questions with a level of difficulty ranging from the graduate level to domain expert current with the state of the art in the field. Human experts reviewed the questions along the criteria of difficulty and specificity, and the model responses along four different criteria: overall quality, specificity, relevance, and accuracy. We ran this benchmark on an instance of OpenAI’s GPT-4o. The responses from the model received a composite quality score of 3.7 on a 1–5 scale, consistent with a passing grade. However, 36% of the questions received at least one below average score. An in-depth analysis of the responses identified at least five instances of suspected hallucination. Finally, we observed statistically significant correlations between the difficulty of the question and the quality of the response, the difficulty of the question and the relevance of the response, the specificity of the question, and the accuracy of the response as graded by the human experts. Furthermore, this emphasizes the need to evaluate LLMs across multiple criteria beyond difficulty or accuracy.

Artificial intelligence↗

NISQ Benchmarking

Test suite of quantum algorithms for Noisy Intermediate Scale Quantum (NISQ) computers. The test suite includes benchmark-style code for quantum volume circuits (QV), fairness sampling circuits, quantum telecloning circuits, and other NISQ benchmark style algorithms on small problems (i.e., up to 100 qubits), such as Variational Quantum Eigensolver (VQE), Hamiltonian Simulation, and Grover unstructured search example circuits. These benchmark-style applications are implemented in quantum software packages, mostly IBM's QISKIT, but may include vendor-specific frameworks, such as PyQuil (for Rigetti) or Q\# for Microsoft, or CirQ (for Google) as the test suite grows with the vendor sample. The test suite also includes numerical simulation code for Quantum Alternating Operator Ansatz (QAOA) algorithms, VQE, Hamiltonian Simulation and search examples. Numerical simulation code simulates quantum computers on classical computers, which is only possible for small problem instances; the implementation framework of choice is typically within Python, using the numpy/scipy libraries as well as extensions to the Julia language.

Pelofske, Elijah↗

High-Performance GMRES Mixed-Precision (HPG-MxP) Benchmark

SAND2024-08539O The High Performance GMRES Mixed-Precision (HPG-MxP) is a benchmark for ranking high-performance supercomputers, allowing use of mixed-precision. Similar to HPCG benchmark, it is designed to profile the computers' capabilities to perform the computational and communication tasks that are commonly found in important classes of real-world applications. At the same time, like HPL-MxP benchmark, it allows the use of mixed-precision arithmetic, while ensuring the double-precision accuracy of the computed solution. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

SciDAC↗

Binder-benchmarking

SAND2025-07593O Binder-benchmarking evaluates the speed and memory impacts of C++, Python, and Matlab code binders. As a repository, it provides a way to locally run computation-based and memory-based benchmark suites on pybind11 and nanobind-based code in a Docker image. The software runs simple-speed and memory benchmarks on primitive navigation and integration exemplar algorithms. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Walker II, Michael [Sandia National Lab. (SNL-CA),↗

Benchmarking operational efficiency in waste collection: Discussion of current approaches and possible alternatives

Efficiency assessment and benchmarking are crucial for managing any organization. However, especially from a regulatory perspective, such efficiency assessment and benchmarking must be unbiased from context-specific issues and should provide an absolute rating, rather than a relative one. The current work reviews the approaches used for performance assessment and benchmarking waste collection services, revealing that the majority are biased and are not absolute, and proposes two alternative context-unbiased and absolute performance indicators, the collection capacity use (CCU) and the segregated waste collection efficiency (SWE). The proposed indicators were calculated for 246 utilities operating in Portugal. The utilities were then ranked accordingly, and their position was compared with the position attained using the equivalent performance indicators in the system currently in use by the Portuguese service regulator. The results reveal ranking differences of over 50 positions and illustrate how misleading the results from context-biased and relative metrics can be.

Sousa, V.↗

Benchmark Analysis

Tremendous progress has been achieved in the development of land models and their inclusion in Earth system models (ESMs). However, we still have very limited knowledge on the performance skills of these land models. This chapter introduces benchmark analysis, which is a procedure to measure performance of models against a set of defined standards. The benchmark analysis includes: (1) defining targeted aspects of model performance to be evaluated; (2) testing model performance in comparison with a set of benchmarks; (3) measuring model performance skill through quantitative metrics; and (4) evaluating model performance and offering suggestions for future model improvement.

Luo, Yiqi↗

Development of a Pulsed Slowing-Down-Time Benchmark of Neutron Thermalization in Graphite

Graphite is a classic neutronic material that has been used as both a reactor reflector and moderator in various nuclear reactor systems. The ability to accurately predict the slowing down and thermalization of neutrons in graphite can have significant implications on the safety and operation of such reactor systems. In reactors, the neutron thermalization process is quantified using the thermal scattering law (TSL) and related cross sections for a given moderator. An ideal approach to assess the validity of TSL data is using benchmark measurement based on the pulsed Slowing-Down-Time technique and its comparison with the appropriate graphite nuclear library. In this work, experimental measurement and computational Monte Carlo simulations were performed to benchmark the slowing down characteristics and thermalization of neutrons in nuclear (reactor-grade) graphite. Given the density of graphite, various graphite libraries were selected for the benchmark analysis.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

bmdrc: Python package for quantifying phenotypes from chemical exposures with benchmark dose modeling

Though chemical exposures are known to potentially have negative impacts on health, including contributing to chronic diseases such as cancer, the quantitative contribution of risk is not fully understood for every chemical. A commonly used approach to quantify levels of risk is to measure the proportion of organisms (such as a total number of zebrafish on a plate or mice in a cage) with abnormal behavioral responses or morphology at increasing concentrations of chemical exposure. A particular challenge with processing the proportional data from these assays is the appropriate estimation of chemical concentration levels that result in malformations or acute toxicity, as these values typically vary between experimental measurements. The recommended approach by the Environmental Protection Agency (EPA) is to fit benchmark dose curves with specific filters and model fitting steps, which are crucial to properly processing the proportional data. Several tools exist for the fitting of benchmark dose response curves, but none are standalone Python libraries built to process both morphological and behavioral data as proportions with all the EPA recommended filters, filter parameters, models, and model parameters. Thus, here we present the benchmark dose response curve (bmdrc) Python library, which was built to closely follow these EPA guidelines with helpful visualizations of filters and fitted model curves, and reports for reproducibility purposes. bmdrc is open-source and has demonstrated utility as a support package to an existing web portal for information on chemicals (https://srp.pnnl.gov). Our package will support any toxicology analysis where the response is a proportional value at increasing levels of a concentration of a chemical or chemical mixture.

Superfund↗

Benchmark Problems of the Geothermal Technologies Office Code Comparison Study

A diverse suite of numerical simulators is currently being applied to predict or understand the performance of enhanced geothermal systems (EGS). To build confidence and identify critical development needs for these analytical tools, the United States Department of Energy, Geothermal Technologies Office has sponsored a Code Comparison Study (GTO-CCS), with participants from universities, industry, and national laboratories. A principal objective for the study was to create a community forum for improvement and verification of numerical simulators for EGS modeling. Teams participating in the study were those representing U.S. national laboratories, universities, and industries, and each team brought unique numerical simulation capabilities to bear on the problems. Two classes of problems were developed during the study, benchmark problems and challenge problems. The benchmark problems were structured to test the ability of the collection of numerical simulators to solve various combinations of coupled thermal, hydrologic, geomechanical, and geochemical processes. This class of problems was strictly defined in terms of properties, driving forces, initial conditions, and boundary conditions. Study participants submitted solutions to problems for which their simulation tools were deemed capable or nearly capable. Some participating codes were originally developed for EGS applications whereas some others were designed for different applications but can simulate processes similar to those in EGS. Solution submissions from both were encouraged. In some cases, participants made small incremental changes to their numerical simulation codes to address specific elements of the problem, and in other cases participants submitted solutions with existing simulation tools, acknowledging the limitations of the code. The challenge problems were based on the enhanced geothermal systems research conducted at Fenton Hill, near Los Alamos, New Mexico, between 1974 and 1995. The problems involved two phases of research, stimulation, development, and circulation in two separate reservoirs. The challenge problems had specific questions to be answered via numerical simulation in three topical areas: 1) reservoir creation/stimulation, 2) reactive and passive transport, and 3) thermal recovery. Whereas the benchmark class of problems were designed to test capabilities for modeling coupled processes under strictly specified conditions, the stated objective for the challenge class of problems was to demonstrate what new understanding of the Fenton Hill experiments could be realized via the application of modern numerical simulation tools by recognized expert practitioners.

15 GEOTHERMAL ENERGY↗

Development and Experimental Benchmark of Simulations to Predict Used Nuclear Fuel Cladding Temperatures during Drying and Transfer Operations

Radial hydride formation in high-burnup used fuel cladding has the potential to radically reduce its ductility and suitability for long-term storage and eventual transport. To avoid this formation, the maximum post-reactor temperature must remain sufficiently low to limit the cladding hoop stress, and so that hydrogen from the existing circumferential hydrides will not dissolve and become available to re-precipitate into radial hydrides under the slow cooling conditions during drying, transfer and early dry-cask storage. The objective of this research is to develop and experimentallybenchmark computational fluid dynamics simulations of heat transfer in post-pool-storage drying operations, when high-burnup fuel cladding is likely to experience its highest temperature. These benchmarked tools can play a key role in evaluating dry cask storage systems for extended storage of high-burnup fuels and post-storage transportation, including fuel retrievability. The benchmarked tools will be used to aid the design of efficient drying processes, as well as estimate variations of surface temperatures as a means of inferring helium integrity inside the canister or cask. This work will be conducted effectively because the principal investigator has experience developing these types of simulations, and has constructed a test facility that can be used to benchmark them.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Integrated FHR technology development: Tritium management, materials testing, salt chemistry control, thermal hydraulics and neutronics, associated benchmarking and commercial basis

This report summarizes the results of a 3-year Integrated Research Project (IRP) sponsored by the U.S. Department of Energy to address major technical challenges in the development of the Fluoride-salt-cooled High-temperature Reactor (FHR). The IRP universities included the Massachusetts Institute of Technology (MIT), the University of California at Berkeley (UCB), the University of Wisconsin (UW), and the University of New Mexico (UNM). The FHR is a new reactor concept less than 20 years old that combines (1) a clean fluoride salt coolant, (2) the graphite-matrix coated-particle fuel originally developed for High- Temperature Gas-Cooled Reactors and (3) passive decay heat removal systems from sodium fast reactors. The base-line design is a pebble-bed FHR that delivers heat between 600 to 700°C with a 7 Li 2 BeF 4 (flibe) salt coolant to the power cycle or industrial users. The Executive Summary describes what was accomplished and summarizes major conclusions. The main report contains stand-alone sections in each major area of work. Highlights are summarized below. Tritium. The coolant salt produces tritium necessitating understanding and control of tritium. Experimental work included measuring hydrogen uptake on carbon, tritium uptake during irradiation of graphite in 700°C salt and the permeability of tritium through metals with and without coatings. An FHR systems model was developed to predict tritium behavior and enable design of tritium control systems. The leading candidate for tritium removal from the flibe salt is carbon bed outside the reactor core. Thermal hydraulics and Neutronics. A 10-meter high FHR integral effects test facility, using Dowtherm A®, has been built and used to simulate the integral response of a representative FHR to improve understanding of FHR thermal hydraulics and validate thermal hydraulics codes used for the FHR. Neutronic analysis included code-to-code benchmarking, multiphysics modeling, uncertainty analysis for nuclear data uncertainties and shielding analysis. Twisted-tube heat exchangers were examined that may provide significant cost reductions because of their improved heat-transfer performance with liquid salts. The impacts on safety and design caused by uncertainties in physical properties of the salt were assessed with recommendations on where added physical property measurements are needed. Work began on understanding radiative heat transfer that becomes important in salt systems at higher temperatures. Materials. Static corrosion tests were conducted on graphite and materials in 700°C salt -- both in the UW laboratory and in the MIT reactor. In addition to static corrosion testing, a circulating salt corrosion loop was built and is operating. Methods to measure salt properties including redox were developed. The presence of carbon increases corrosion. With good redox control, the evidence indicates stainless steel can be used as a material of construction. Benchmarking. An additional series of workshops where held to enable benchmarking and comparison of results between different investigators. Commercialization. The FHR delivers higher average-temperature (~650°C) heat to the power cycle or industry than other reactors. This enables power cycles with lower-cost heat storage for variable electricity to the grid while the reactor operates at base-load -- providing a major competitive advantage by maximizing revenue from electricity sales. In October, 2016, a new startup company named Kairos Power was formed, and opened its first office in Jack London Square, Oakland, California, in January 2017. Kairos Power exists largely due to the work that was accomplished by this IRP and the previous DOE-funded FHR IRP that performed research on FHR technology from 2012 to 2015. In September, 2018, as this IRP is concluding, Kairos Power was staffed at 60 full time employees, working to develop technology derived from this IRP. Many of these employees worked with the IRP before joining Kairos Power 1 (https://kairospower.com). Other students and researchers from this IRP have joined other startup companies, including the Terrapower Molten Chloride Fast Reactor and the Terrestrial Energy development efforts.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Nuclear Data and Benchmarking Program: Nuclear Data Performance Assessment for Advanced Reactors

Over the last several Evaluated Nuclear Data File (ENDF)/B releases, many updates have been introduced to the nuclear data libraries that can significantly change computational results. These changes can be particularly important for advanced reactor concepts, which are not as thoroughly investigated as those used in light water reactor (LWR) systems. Therefore, a performance assessment of the ENDF/B libraries using the SCALE 6.2.3 code package was conducted under the auspices of the U.S., Department of Energy Office of Nuclear Energy (DOE-NE). The following systems relevant to the advanced reactor community were chosen as subject for investigation in this report: sodium cooled fast reactor systems, graphite moderated high temperature gas-cooled reactors, and several molten salt reactor models. At first, the similarity of the integral performance of the ENDF/B-VII.1 and ENDF/B-VIII.0 nuclear data libraries caused by compensating errors in important differential data was demonstrated. In calculations for a shipping container of high-assay low enriched uranium, the data of individual isotopes were systematically swapped between the two ENDF/B libraries. An eigenvalue difference of up to 450 pcm was found due to the use of 235 U and 238 U from one library and 1 H and 16 O from the other library. This clearly demonstrates a cross-correlation between reaction data sets of different isotopes within a library that should be reported in the evaluations. For sodium-cooled fast reactor (SFR) fuel assemblies, significant eigenvalue (k inf ) differences (200– 450 pcm) were found between calculations using the 2011 ENDF/B-VII.1 data and 2018 ENDF/B-VIII.0 data. These differences were mainly caused by updates of 238 U and 239 Pu neutron cross sections. Results from the multigroup (MG) calculations further revealed that the group structure of the applied MG library strongly influences the MG bias due to the importance of an appropriate energy resolution of the resonances in higher energy ranges. The application of previously used MG libraries requires new verification for advanced reactor simulation in comparisons with reference continuous-energy calculations. Nuclear data uncertainty analyses of these SFR systems resulted in eigenvalue uncertainties between 1,400 and 1,800 pcm, which are three to four times higher than corresponding uncertainties in light water reactor (LWR) systems. The main contributor to this uncertainty was found to be inelastic scattering on 238 U, which shows an uncertainty of up to 50% in the fast energy range. Other important contributors are the scattering reactions of 56 Fe and 23 Na which have so far not appeared in LWR analysis. Since the largest contribution to the eigenvalue uncertainty of SFR systems is coming from scattering reactions, it is expected that uncertainties in the angular scattering distributions also have a significant impact. Those uncertainties are, however, not available for the majority of nuclides, and the capabilities to determine sensitivities to this data are currently underdeveloped. For two graphite moderated high temperature gas-cooled reactor (HTGR) benchmarks, only the ENDF/BVII. 1 calculations resulted in consistent eigenvalues (k eff ) with the corresponding experimental measurements. Eigenvalue differences of about 1,000 pcm were found between the 2006 ENDF/B-VII.0 data and the 2011 ENDF/B-VII.1 data, primarily due to an updated carbon capture cross section in the thermal energy range in the later ENDF/B release. The ENDF/B-VIII.0 eigenvalue was larger than the ENDF/B-VII.1 eigenvalue by about 300 pcm mainly due to updates in multiple 235 U neutron cross sections, with offsetting updates in the 238 U cross sections. With ENDF/B-VIII.0, graphite can be modeled as perfect crystal or with two different porosities. The choice of the graphite evaluation can have a significant influence of the eigenvalue. For a HTGR benchmark, maximum eigenvalue differences as high as 650 pcm due to different porosities were observed. More detailed studies of the impact of the porosity are necessary while considering that the graphite porosity can vary between the used materials and changes as a function of neutron fluence. xii The uncertainty of the HTGR eigenvalues due to nuclear data uncertainties was found between 500 and 600 pcm, with the top contributor being the neutron multiplicity of 235 U. A gap in the form of missing uncertainties in graphite thermal scattering data, that might have significant impact on HTGR reactors, was identified. This gap is being investigated by the DOE-NE Nuclear Data and Benchmarking Program, as documented in a separate report The fast spectrum molten salt reactor calculations revealed similar comparisons as for the SFR assemblies. The eigenvalue uncertainties were significantly influenced by inelastic scattering on 238 U, while the impact of angular scattering distributions is unknown. For the graphite and zirconium hydride moderated system, eigenvalue uncertainties of up to 700 pcm were observed, requiring in depth analyses of the cross sections and corresponding uncertainties of the included materials. In general, it was observed that the flux spectra of the various molten salt reactor systems show significant differences, even between a fresh state and a depleted state. The choice of the energy group structure of MG calculations is therefore highly relevant; the applicability of previously used MG libraries needs to be verified. Due the unavailability of pin power measurements of advanced reactor systems, pin power calculations with both ENDF/B-VII.1 and VIII.0 data were performed for a light water reactor and compared to corresponding measurements. Both calculations show good agreement with the measurements. A comparison between the ENDF/B-VII.1 and VIII.0 results revealed small differences, mostly in the range of about 0.2%.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Critical Experiment Benchmark Results using MCNP6.2 Unstructured Mesh

The purpose of this report is to document the results of five critical experiment benchmarks modeled using the unstructured mesh (UM) geometry capability in MCNP®1 code, version 6.2. LANL currently has a vast critical experiment benchmark library with over 1100 cases, modeled using constructive solid geometry (CSG) and packaged with Whisper-1.1. Validation efforts for the purposes of nuclear criticality safety typically focus on benchmark libraries built using CSG models. Results of UM geometry models are compared with CSG results and the critical experiment k-effective from the ICSBEP handbook. All computed k-effective results in this study are well within 1% of the experimental result.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Benchmark problems for the Me soscale M ultiphysics Ph ase F i eld S imulator (MEMPHIS)

This report details the current benchmark results to verify, validate and demonstrate the capabilities of the in-house multi-physics phase-field modeling framework Me soscale M ultiphysics Ph ase F i eld S imulator (MEMPHIS) developed at the Center for Integrated Nanotechnologies (CINT). MEMPHIS is a general phase-field capability to model various nanoscience and materials science phenomena related to microstructure evolution. MEMPHIS has been benchmarked against a suite of reported 'classical' phase-field benchmark problems to verify and validate the correctness, accuracy and precision of the models and numerical methods currently implemented into the code.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗