Search NASA⌕ Search

SEARCH · Search NASA

Results for “performance benchmarking”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

ChemGraph as an agentic framework for computational chemistry workflows

Atomistic simulations are essential in chemistry and materials science but remain challenging to run due to the expert knowledge required for the setup, execution, and validation stages of these calculations. We present ChemGraph, an agentic framework powered by artificial intelligence and state-of-the-art simulation tools to streamline and automate computational chemistry and materials science workflows. ChemGraph leverages graph neural network-based foundation models for accurate yet computationally efficient calculations and large language models (LLMs) for natural language understanding, task planning, and scientific reasoning to provide an intuitive and interactive interface. We evaluate ChemGraph across 13 benchmark tasks and demonstrate that smaller LLMs (GPT-4o-mini, Claude-3.5-haiku, Qwen-2.5-14B) perform well on simple workflows, while more complex tasks benefit from using larger models. Importantly, we show that decomposing complex tasks into smaller subtasks through a multi-agent framework enables GPT-4o to reach perfect accuracy and smaller LLMs to match or exceed single-agent GPT-4o's performance in these benchmarks.

Computational chemistry↗
DOE OSTI2022

Health Physics Research Reactor Criticality Accident Alarm System Benchmark Overview

From the countless critical experiments performed in the world during the past century, high-quality integral benchmarks experiments have been collected and gathered into the International Handbook of Evaluated Criticality Safety Benchmark Experiments (ICSBEP Handbook), managed by the International Criticality Safety Benchmark Evaluation Project (ICSBEP) Working Group. This information preservation and dissemination effort is crucial for reactor licensing as well as criticality and radiation transport modeling validation. This summary reports on the status of a tentative benchmark addition to the ICSBEP Handbook. The proposed benchmark arises from legacy operation data of the Oak Ridge National Laboratory (ORNL) Health Physics Research Reactor (HPRR). The HPRR was a small, unmoderated, unshielded fast burst reactor that was used for research in health physics and radiobiology as well as teaching and training. As part of a comprehensive investigation of the available HPRR operation data and characteristics, different possibilities for use of the valuable results were studied. A critical experiment benchmark evaluation was performed, analyzing data coming from sub-critical and critical operation of the HPRR during operator training, steady-state irradiation of samples and before critical bursts. The results of the evaluation do not satisfy for the ICSBEP standards as the benchmark relative standard uncertainty is of about 4% for k eff , and the relative difference between sample calculations and expected k eff results is of about 1.5%. Due to those unsatisfactory results, it was decided not to pursue critical experiments evaluation of the HPRR presently and to focus instead on shielding type data for the creation of a criticality accident alarm system (CAAS) and shielding category benchmark, which is currently very scarce in the ICSBEP handbook—especially concerning critical, pulsed assembly, or reactor operation data. Several dosimetry and shielding experiments from HPRR burst operation were evaluated, with different benchmark metrics as sulfur fluence, Element 57 dose, or neutron fluence at different distances and under different shield materials conditions. An evaluation focusing on the Element 57 neutron dose as a benchmark metric was submitted to the ICSBEP Technical Review Group (TRG) meeting in October 2021, and the inclusion of the evaluation in the ICSBEP Handbook was deferred. The main change proposed by the international experiment evaluation experts is to use the neutron fluence measured by Bonner spheres as a benchmark metric. This represents a quantity closer to that actually measured by the experimentalists of the HPRR compared to the Element 57 dose, which adds another step of data transformation, thus potentially adding uncertainty to the benchmark. The evaluation has been updated and will be submitted to the 2022 ICSBEP TRG meeting for inclusion in the 2023 edition of the ICSBEP Handbook. The evaluation is performed using the KENO and MAVRIC combination from the SCALE 6.2.4 code suite which was previously used in similar CAAS benchmarks to allow for the use of variance reduction techniques.

61 RADIATION PROTECTION AND DOSIMETRY↗
DOE OSTI2021

Case Study of Using Kokkos and SYCLs Performance-Portable Frameworks for Milc-Dslash Benchmark on NVIDIA, AMD and Intel GPUs

Six of the top ten supercomputers in the TOP500 list from June 2021 rely on NVIDIA GPUs to achieve their peak compute bandwidth. With the announcement of Aurora, Frontier, and El Capitan, Intel and AMD have also entered the domain of providing GPUs for scientific computing. A consequence of the increased diversity in the GPU landscape is the emergence of portable programming models such as Kokkos, SYCL, OpenCL, and OpenMP, which allow application developers to maintain a single-source code across a diverse range of hardware architectures. While the portable frameworks try to optimize the compute resource usage on a given architecture, it is the programmers responsibility to expose parallelism in an application that can take advantage of thousands of processing elements available on GPUs. In this paper, we introduce a GPU-friendly parallel implementation of Milc-Dslash that exposes multiple hierarchies of parallelism in the algorithm. Milc-Dslash was designed to serve as a benchmark with highly optimized matrix-vector multiplications to measure the resource utilization on the GPU systems. The parallel hierarchies in the Milc-Dslash algorithm are mapped onto a target hardware using Kokkos and SYCL programming models. We present the performance achieved by Kokkos and SYCL implementations of Milc-Dslash on NVIDIA A100 GPU, AMD MI100 GPU, and Intel Gen9 GPU. Additionally, we compare the Kokkos and SYCL performances with those obtained from the versions written in CUDA and HIP programming models on NVIDIA A100 GPU and AMD MI100 GPU, respectively.

Dufek, Amanda S↗
DOE OSTI2020

PRISMS-PF: A general framework for phase-field modeling with a matrix-free finite element method

Abstract A new phase-field modeling framework with an emphasis on performance, flexibility, and ease of use is presented. Foremost among the strategies employed to fulfill these objectives are the use of a matrix-free finite element method and a modular, application-centric code structure. This approach is implemented in the new open-source PRISMS-PF framework. Its performance is enabled by the combination of a matrix-free variant of the finite element method with adaptive mesh refinement, explicit time integration, and multilevel parallelism. Benchmark testing with a particle growth problem shows PRISMS-PF with adaptive mesh refinement and higher-order elements to be up to 12 times faster than a finite difference code employing a second-order-accurate spatial discretization and first-order-accurate explicit time integration. Furthermore, for a two-dimensional solidification benchmark problem, the performance of PRISMS-PF meets or exceeds that of phase-field frameworks that focus on implicit/semi-implicit time stepping, even though the benchmark problem’s small computational size reduces the scalability advantage of explicit time-integration schemes. PRISMS-PF supports an arbitrary number of coupled governing equations. The code structure simplifies the modification of these governing equations by separating their definition from the implementation of the numerical methods used to solve them. As part of its modular design, the framework includes functionality for nucleation and polycrystalline systems available in any application to further broaden the phenomena that can be used to study. The versatility of this approach is demonstrated with examples from several common types of phase-field simulations, including coarsening subsequent to spinodal decomposition, solidification, precipitation, grain growth, and corrosion.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Utilizing ensemble learning for performance and power modeling and improvement of parallel cancer deep learning CANDLE benchmarks

Abstract Machine learning (ML) continues to grow in importance across nearly all domains in modeling to learn from data. Often a tradeoff exists between a model's ability to minimize bias and variance. In this article, we utilize ensemble learning to combine linear, nonlinear, and tree‐/rule‐based ML methods to cope with the bias‐variance tradeoff and result in more accurate models. We use the datasets collected for two parallel cancer deep learning CANDLE benchmarks, NT3 and P1B2, to build performance and power models based on hardware performance counters using single‐object and multiple‐objects ensemble learning to identify the most important counters for improvement on the Cray XC40 Theta at Argonne National Laboratory. Based on the insights from these models, we improve the performance and energy of P1B2 and NT3 by optimizing the deep learning environments TensorFlow, Keras, Horovod, and Python under the huge page size of 8 MB. Experimental results show that ensemble learning not only produces more accurate models but also provides more robust performance counter ranking. We achieve up to 61.15% performance improvement and up to 62.58% energy saving for P1B2 and up to 55.81% performance improvement and up to 52.60% energy saving for NT3 on up to 24,576 cores.

Wu, Xingfu↗
DOE OSTI2022

High-performance and high-fidelity Monte Carlo solutions to the BEAVRS benchmark

The BEAVRS (Benchmark for Evaluation and Validation of Reactor Simulation) benchmark is solved by PRAGMA, the GPU-based continuous energy Monte Carlo code. The resulting solutions are comprised of the detailed simulation results of two cycles, each of which consists of the zero power physics test (ZPPT) and the core depletion calculations. The ZPPT consists of characteristic parameters, such as critical boron concentration (CBC), control rod bank worth, isothermal temperature coefficients, and assembly-wise detector signal, which are compared with measured data provided by the benchmark administration. The core depletion calculations were performed for both the hot full power and the load follow modes, and the comparison was made with the measured or deduced CBCs and assembly-wise detector signals. In the load follow calculations, the operating power history was approximately applied to simulate the real operation as closely as possible. PRAGMA performed the various calculations with a tremendous number of histories ranging up to hundreds of millions per cycle, exploiting GPUs' massively parallel performance. The load follow run time was shorter than 16 hours on a single rack of computing nodes mounded with 24 gaming GPUs with a remarkable agreement with the measurements for most comparisons. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2025

Critical Simulation Pipeline for COG Suites [Poster]

The CRItical Simulation Pipeline (CRISP) is a Python package for automating validation of reactor criticality benchmarks. CRISP supplies COG—a multi-particle radiation transport code maintained by the Nuclear Criticality Safety Division—with a pipeline to calculate k eff performance for 400+ benchmark experiments with 3,400+ configurations from the International Criticality Safety Benchmark Evaluation Project (ICSBEP). The pipeline includes four stages: materials configuration, input card templating, cluster submission, and results analysis. CRISP includes a command-line interface to facilitate user interaction.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2022

A Comparison of Preconstruction and Operational Wake Loss Estimates for Land-Based Wind Plants

Recent studies suggest that biases between wind plant pre-construction energy yield estimates and actual energy production are decreasing over time. However, variability in energy yield prediction accuracy across different projects and wind energy consultants remains high. Wake effects are one of the largest categories comprising the pre-construction energy yield assessment process. To assess the accuracy of wake loss predictions, we compare pre-construction wake loss estimates provided by 8 consultants to the estimated operational wake losses for 10 North American wind plants, as part of the Wind Plant Performance Prediction (WP3) Benchmark project. We estimate operational wake losses using supervisory control and data acquisition (SCADA) data by comparing total wind plant energy production to the potential energy production based on the power produced by freestream wind turbines. In the presentation, we will discuss the overall wake loss prediction bias as well as the project-to-project variability in the prediction accuracy. Further, we will highlight challenges encountered when estimating operational wake losses, including the impact of complex terrain and the presence of neighboring wind plants.

benchmark↗
DOE OSTI2020

Off-design performance of molten salt-driven Rankine cycles and its impact on the optimal dispatch of concentrating solar power systems

This paper presents a model for improving off-design performance predictions for molten salt-driven Rankine power cycles, such as in concentrating solar power tower applications. The model predicts cycle off-design performance under various boundary conditions, including molten salt inlet temperature, mass flow rate, and ambient temperature. The model is validated using industry performance data and benchmarked with results from the literature. A complete concentrating solar power plant, inclusive of solar heliostat field and receiver, is then considered, by implementing the Rankine cycle off-design performance results into the National Renewable Energy Laboratory’s System Advisor Model software, which includes a tool that determines optimal power production schedules. The work improves upon the current System Advisor Model by updating off-design performance characteristics. A case study demonstrates the impact of cycle off-design behavior on annual performance for a stand-alone concentrating solar power system and a concentrating solar power-photovoltaic hybrid system. In addition, we demonstrate how cycle off-design performance influences optimal operator dispatch decisions and, thereby, overall system design and economics. We conclude that off-design cycle performance impacts “optimal” sub-system sizing, especially for a concentrating solar power-photovoltaic hybrid configuration in which concentrating solar power must dispatch in conjunction with photovoltaic generation.

14 SOLAR ENERGY↗
DOE OSTI2022

Evaluating Unified Memory Performance in HIP

Heterogeneous unified memory management between a CPU and a GPU is a major challenge in GPU computing. Recently, unified memory (UM) has been supported by software and hardware components on AMD computing platforms. The support could simplify the complexities of memory management. In this paper, we attempt to have a better understanding of UM by evaluating the performance of UM programs on an AMD MI100 GPU. More specifically, we evaluate data migration using UM against other data transfer techniques for the overall performance of an application, assess the impacts of three commonly used optimization techniques on the kernel execution time of a vector add sample, and compare the performance and productivity of selected benchmarks with and without UM. The performance overhead associated with UM is not trivial, but it can improve programming productivity by reducing lines of code for scientific applications. We aim to present early results and feedback on the UM performance to the vendor.

Jin, Zheming↗
DOE OSTI2024

On the emerging potential of quantum annealing hardware for combinatorial optimization

Abstract Over the past decade, the usefulness of quantum annealing hardware for combinatorial optimization has been the subject of much debate. Thus far, experimental benchmarking studies have indicated that quantum annealing hardware does not provide an irrefutable performance gain over state-of-the-art optimization methods. However, as this hardware continues to evolve, each new iteration brings improved performance and warrants further benchmarking. To that end, this work conducts an optimization performance assessment of D-Wave Systems’ Advantage Performance Update computer, which can natively solve sparse unconstrained quadratic optimization problems with over 5,000 binary decision variables and 40,000 quadratic terms. We demonstrate that classes of contrived problems exist where this quantum annealer can provide run time benefits over a collection of established classical solution methods that represent the current state-of-the-art for benchmarking quantum annealing hardware. Although this work does not present strong evidence of an irrefutable performance benefit for this emerging optimization technology, it does exhibit encouraging progress, signaling the potential impacts on practical optimization tasks in the future.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2022

Out of Distribution Detection with Neural Network Anchoring

This is code to reproduce and build on OOD detection from the paper "Out of Distribution Detection with Neural Network Anchoring". Our goal here is to exploit heteroscedastic temperature scaling as a calibration strategy for out of distribution (OOD) detection. Heteroscedasticity here refers to the fact that the optimal temperature parameter for each sample can be different, as opposed to conventional approaches that use the same value for the entire distribution. To enable this, we propose a new training strategy called anchoring that can estimate appropriate temperature values for each sample, leading to state-of-the-art OOD detection performance across several benchmarks. Using NTK theory, we show that this temperature function estimate is closely linked to the epistemic uncertainty of the classifier, which explains its behavior. In contrast to some of the best-performing OOD detection approaches, our method does not require exposure to additional outlier datasets, custom calibration objectives, or model ensembling. Through empirical studies with different OOD detection settings - far OOD, near OOD, and semantically coherent OOD - we establish a highly effective OOD detection approach.

Thiagarajan, Jayaraman↗
DOE OSTI2022

ORNL Neutron Cross Section Measurements of 90 Zr

Nuclear criticality modeling and simulations rely on the quality of the existing evaluated nuclear data libraries such as Evaluated Nuclear Data File (ENDF)/B, the Joint Evaluated Fission and Fusion (JEFF) nuclear data library, or the Japanese Evaluated Nuclear Data Library (JENDL). In some cases, the cross-section evaluations of those libraries were found to be deficient in describing criticality benchmarks accurately. More than two decades ago, the US Nuclear Criticality Safety Program (NCSP) established a Nuclear Data (ND) task which encompassed experiments and evaluations. In response to this, the Oak Ridge National Laboratory (ORNL) formed a Nuclear Criticality and Data group which performed ND experiments, data analysis, and evaluations to produce ENDF files for the ND libraries as identified in the NCSP Five-Year Plan. Before being submitted to the ENDF library, files were processed and tested for performance by running benchmark calculations. This procedure was centralized in the ORNL group and is now often referred to as the ND pipeline. NCSP collaborates with the Joint Research Center (JRC) of the European Commission in Geel, Belgium, to perform high-resolution neutron-induced cross section measurements at the Geel Linear Accelerator (GELINA). The objective is to address emerging ND problems in criticality calculations. Difficulties with ND include insufficient neutron energy range, missing covariances, and previously unrecognized inaccuracies with experiments. New neutron total and capture cross sections of 90 Zr in the neutron energy range from 100 eV to several hundred keV were recently performed. These measured data will be used, together with existing high-resolution transmission data from a metallic 90 Zr sample, to improve representation of the cross sections.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Creep and Fatigue Characterization of High Strength Alloy Thin Sections in Advanced CO2 Heat Exchangers

The objective of this work was to characterize and model elevated temperature creep and fatigue behavior for thin sheet and foil forms of gamma-prime strengthened alloys in wrought form and as-processed folded and brazed constructions. This work was motivated by the demanding temperature and pressure service conditions of the GEN3 Concentrated Solar Power (CSP) and supercritical CO2 (sCO2) power cycle working fluid. More specifically, the possibility of leveraging the superior creep strength of gamma-prime alloys in folded-fin and brazed-plate heat exchanger constructions. Gamma-prime alloys represent a step-change in raw-material strength over solid-solution strengthened alloys. And the folded-fin and brazed-plate heat exchanger architecture is lightweight and leverages cost-effective material stock forms. The investigation contained two parallel paths. (1) The first is referred to as a fundamental investigation where Oak Ridge National Laboratory conducts uniaxial creep testing on thin sheet and foil in wrought form. This effort aimed to serve as a benchmark against a relatively sparse existing database and a baseline comparison for path number 2. (2) The second path is referred to as the practical investigation where Brayton Energy manufactures plate-fin heat exchangers and performs pressurized creep and fatigue testing. This effort aimed to de-risk heat exchanger manufacturing process for service under sCO2 CSP conditions. A total of 14 uniaxial creep tests were completed using Haynes 282 thin sheet and foil. A variety of heat treatments were specified to coincide with path number 2. Baseline metallography of test samples and creep strength performance are contained. Benchmarks relatively to existing thick-form Haynes 282 are made, as well as to other thin-form Nickel-based superalloys. Description of a wrought-form modeling approach for thin gamma-prime alloys is also discussed. A total of 11 pressurized creep and fatigue tests were completed successfully with Haynes 282 heat exchanger prototypes. Manufacturing processing details, testing details, testing results, and failure analysis are discussed. Additionally, creep modeling techniques to predict failure are discussed, and modeling to support technological-to-market. In conclusion, Haynes 282 foils were demonstrated to yield rupture two-to-three orders of magnitude higher than Haynes 230 foils under similar conditions. And the manufactured heat exchanger prototypes demonstrated strength similar to the wrought constituents. Both of which contribute to elevated performance potential or cost savings in practice. Discussion is included.

14 SOLAR ENERGY↗
DOE OSTI2024

Benchmark Calculation for the Peach Bottom Unit 2 Cycles 1-3 Using the SCALE 6.3/Polaris–PARCS v3.4.2 Code Package

In this study, benchmark calculations were performed for Peach Bottom Unit 2 cycles 1–3 to validate the SCALE 6.3/Polaris–PARCS v3.4.2 with the ENDF/B-VII.1 AMPX 56-group library by comparing the simulated results with the measured data. The benchmark results will be used to evaluate uncertainties of the SCALE/Polaris–PARCS code package for boiling water reactor physics analysis for key nuclear parameters such as reactivity and assembly power peaking factors. This report details plant and fuel design specifications and input data for SCALE/Polaris, GenPMAXS and PARCS. Additionally, detailed information is provided for all the input and output files produced for the benchmark calculations. The benchmark results were summarized such that they can be used to evaluate uncertainties with other benchmark results for key nuclear parameters.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2024

Benchmark Calculation for Turkey Point Unit 3 Cycles 1-3 Using the SCALE 6.3/Polaris–PARCS v3.4.2 Code Package

Benchmark calculations were performed for Turkey Point Unit 3 cycles 1–3 to validate the SCALE 6.3/Polaris–PARCS v3.4.2 code with the ENDF/B–VII.1 56–group library by comparing the simulated results with the measured data. The benchmark results will be used in evaluating the SCALE/Polaris–PARCS code package’s uncertainties for pressurized water reactor physics analysis. That future analysis will include key nuclear parameters such as reactivity, control bank worth, temperature coefficients, and pin and assembly power peaking factors. The present document details plant and fuel design specifications and input data for SCALE/Polaris, GenPMAXS, and PARCS. Additional details are provided with respect to the input and output files produced for the benchmark calculations. The benchmark results are summarized such that they can be used in evaluating uncertainties with other benchmark results for key nuclear parameters.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2023

Low-loss interconnects for modular superconducting quantum processors

Low-loss superconducting aluminium cables and on-chip impedance transformers can be used to link qubit modules and create superconducting quantum computing networks with high-fidelity intermodule state transfer. Scaling is now a key challenge in superconducting quantum computing. One solution is to build modular systems in which smaller-scale quantum modules are individually constructed and calibrated and then assembled into a larger architecture. This, however, requires the development of suitable interconnects. Here we report low-loss interconnects based on pure aluminium coaxial cables and on-chip impedance transformers featuring quality factors of up to 8.1 x 10 5 , which is comparable with the performance of our transmon qubits fabricated on a single-crystal sapphire substrate. We use these interconnects to link five quantum modules with intermodule quantum state transfer and Bell state fidelities of up to 99%. To benchmark the overall performance of the processor, we create maximally entangled, multiqubit Greenberger-Horne-Zeilinger states. The generated intermodule four-qubit Greenberger-Horne-Zeilinger state exhibits 92.0% fidelity. We also entangle up to 12 qubits in a Greenberger-Horne-Zeilinger state with 55.8 ± 1.8% fidelity, which is above the genuine multipartite entanglement threshold of 1/2.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2022

DMC-ICE13 : Ambient and high pressure polymorphs of ice from diffusion Monte Carlo and density functional theory

Ice is one of the most important and interesting molecular crystals, exhibiting a rich and evolving phase diagram. Recent discoveries mean that there are now 20 distinct polymorphs; a structural diversity that arises from a delicate interplay of hydrogen bonding and van der Waals dispersion forces. This wealth of structures provides a stern test of electronic structure theories, with Density Functional Theory (DFT) often not able to accurately characterize the relative energies of the various ice polymorphs. Thanks to recent advances that enable the accurate and efficient treatment of molecular crystals with Diffusion Monte Carlo (DMC), we present here the DMC-ICE13 dataset; a dataset of lattice energies of 13 ice polymorphs. This dataset encompasses the full structural complexity found in the ambient and high-pressure molecular ice polymorphs, and when experimental reference energies are available, our DMC results deliver sub-chemical accuracy. Using this dataset, we then perform an extensive benchmark of a broad range of DFT functionals. Of the functionals considered, revPBE-D3 and RSCAN reproduce reference absolute lattice energies with the smallest error, while optB86b-vdW and SCAN+rVV10 have the best performance on the relative lattice energies. Our results suggest that a single functional achieving reliable performance for all phases is still missing, and that care is needed in the selection of the most appropriate functional for the desired application. The insights obtained here may also be relevant to liquid water and other hydrogen-bonded and dispersion-bonded molecular crystals.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗