SEARCH · Search NASA

Results for “model benchmarking”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16

All sources NASA NTRS DOE OSTI

NASA NTRS1998

Predicting Cost/Performance Trade-offs For Whitney: A Commodity Computing Cluster

Recent advances in low-end processor and network technology have made it possible to build a "supercomputer" out of commodity components. We develop simple models of the NAS Parallel Benchmarks version 2 (NPB 2) to explore the cost/performance trade-offs involved in building a balanced parallel computer supporting a scientific workload. By measuring single processor benchmark performance, network latency, and network bandwidth, and using closed form expressions detailing the number and size of messages sent by each benchmark, our models predict benchmark performance to within 30%. A comparison based on total system cost reveals that current commodity technology (200 MHz Pentium Pros with 100baseT Ethernet) is well balanced for the NPBs up to a total system cost of around $ 1,000,000.

Becker, Jeffrey C.↗

DOE OSTI2024

Benchmark High-Fidelity EMT Models for Power Grid with PV Plants

In recent times electromagnetic transient (EMT) modeling tools have been identified as one of the most important requirements in replicating, analyzing, and investigating the dynamics of the power grid with photovoltaic (PV) plants. However, there are no benchmark models for power grid with PVs to investigate emerging challenges with higher penetration of PVs (like trips and momentary cessations during faults from a region far away). To this end, in this paper, synthetic benchmark high-fidelity EMT dynamic models of power grid with large-scale PV plants are presented. The models are developed in PSCAD and PSCAD/Fortran. Simulation results for different use cases (events) and scenarios are presented.

Marthi, Phani Ratna Vanamali↗

DOE OSTI2025

Temporal Forecasting of Distributed Temperature Sensing in a Thermal Hydraulic System With Machine Learning and Statistical Models

We benchmark performance of long-short term memory (LSTM) network machine learning model and autoregressive integrated moving average (ARIMA) statistical model in temporal forecasting of distributed temperature sensing (DTS). Data in this study consists of fluid temperature transient measured with two co-located Rayleigh scattering fiber optic sensors (FOS) in a forced convection mixing zone of a thermal tee. We treat each gauge of a FOS as an independent temperature sensor. We first study prediction of DTS time series using Vanilla LSTM and ARIMA models trained on prior history of the same FOS that is used for testing. The results yield maximum absolute percentage error (MaxAPE) and root mean squared percentage error (RMSPE) of 1.58% and 0.06% for ARIMA, and 3.14% and 0.44% for LSTM, respectively. Next, we investigate zero-shot forecasting (ZSF) with LSTM and ARIMA trained on history of the co-located FOS only, which is advantageous when limited training data is available. The ZSF MaxAPE and RMSPE values for ARIMA are comparable to those of the Vanilla use case, while the error values for LSTM increase. We show that in ZSF, performance of LSTM network can be improved by training on most correlated gauges between the two FOS, which are identified by calculating the Pearson correlation coefficient. The improved ZSF MaxAPE and RMSPE for LSTM are 4.4% and 0.33%, respectively. Performance of ZSF LSTM can be further enhanced through transfer learning (TL), where LSTM is re-trained on a subset of the FOS that is the target of forecasting. We show that LSTM pre-trained on correlated dataset and re-trained on 30% of testing target dataset achieves MaxAPE and RMSPE values of 2.32% and 0.28%, respectively.

ARIMA↗

DOE OSTI2024

Open‐Source Anaerobic Digestion Modeling Platform, Anaerobic Digestion Model No. 1 Fast (ADM1F)

An open‐source modeling platform, called Anaerobic Digestion Model No. 1 Fast (ADM1F), is introduced to achieve fast and numerically stable simulations of anaerobic digestion processes. ADM1F is compatible with an iPython interface to facilitate model configuration, simulation, data analysis, and visualization. Faster simulations and more stable results are accomplished by implementing an advanced open‐source library of numerical methods called Portable Extensive Toolkit for Scientific Computation (PETSc) to solve the ADM1 system of equations. Leveraging PETSc, ADM1F can consistently complete a steady‐state simulation under 0.2 s, over 99% faster than a benchmark ADM1 model implemented with MATLAB while achieving agreement of model outputs within 1% of those obtained with the benchmark model. For dynamic simulations, however, ADM1F has a computational speed advantage only when the influent characteristics update more frequently than every 4 h. The ability of ADM1F to be useful as a tool to study anaerobic digestion systems is demonstrated through two example implementations of ADM1F: (1) a two‐phase co‐digestion scenario evaluating the impact of the organic loading rate and the substrate composition on reactor performance and stability, and (2) a conventional digester scenario assessing the effectiveness of recovery strategies after disruptions that led to instability. These examples demonstrate how the high simulation speed and the convenience of the iPython interface allow ADM1F to complete complex analyses within minutes, much faster than computational strategies currently reported in the literature.

anaerobic co-digestion↗

DOE OSTI2025

LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators

Large Language Models (LLMs) have propelled groundbreaking advancements across several domains and are commonly used for text generation applications. However, the computational demands of these complex models pose significant challenges, requiring efficient hardware acceleration. Benchmarking the performance of LLMs across diverse hardware platforms is crucial to understanding their scalability and throughput characteristics. We introduce LLM-Inference-Bench, a comprehensive benchmarking suite to evaluate the hardware inference performance of LLMs. We thoroughly analyze diverse hardware platforms, including GPUs from Nvidia and AMD and specialized AI accelerators, Intel Habana and SambaNova. Our evaluation includes several LLM inference frameworks and models from LLaMA, Mistral, and Qwen families with 7B and 70B parameters. Our benchmarking results reveal the strengths and limitations of various models, hardware platforms, and inference frameworks. We provide an interactive dashboard to help identify configurations for optimal performance for a given hardware platform.

Chitty-Venkata, Krishna Teja↗

DOE OSTI2025

Model-agnostic search for dijet resonances with anomalous jet substructure in proton–proton collisions at $\sqrt{s}$ = 13 TeV

This paper presents a model-agnostic search for narrow resonances in the dijet final state in the mass range 1.8-6 TeV. The signal is assumed to produce jets with substructure atypical of jets initiated by light quarks or gluons, with minimal additional assumptions. Search regions are obtained by utilizing multivariate machine-learning methods to select jets with anomalous substructure. A collection of complementary anomaly detection methods - based on unsupervised, weakly supervised, and semisupervised algorithms - are used in order to maximize the sensitivity to unknown new physics signatures. These algorithms are applied to data corresponding to an integrated luminosity of 138 fb -1 , recorded by the CMS experiment at the LHC, at a center-of-mass energy of 13 TeV. No significant excesses above background expectations are seen. Exclusion limits are derived on the production cross section of benchmark signal models varying in resonance mass, jet mass, and jet substructure. Many of these signatures have not been previously sought, making several of the limits reported on the corresponding benchmark models the first ever. When compared to benchmark inclusive and substructure-based search strategies, the anomaly detection methods are found to significantly enhance the sensitivity to a variety of models.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2023

Experiment Design and Preparation for a Shielding Benchmark Utilizing Godiva-IV

An experiment is currently being designed to provide a high-quality shielding benchmark for Criticality Accident Alarm System (CAAS) modeling. Previous benchmarks have suffered from uncertainty primarily due to two factors. The first factor is radiation that is not directly coming from the source, also known as room return. Room return is notoriously difficult to account for in experimental data and to model in benchmarks. The second factor is uncertainty associated with the source term itself. In order to reduce these two sources of uncertainty, this experiment will utilize a room return shield [1] that will reduce the effect of room return and previous work performed to ensure the reproducibility of the source term. The experiment will use the Godiva IV [2] assembly located at the National Criticality Experiments Research Center (NCERC) to provide a neutron source representative of a criticality accident. Previous experiments [3]-[4] have shown that Godiva IV is capable of producing both bursts and steady-state emissions of equal magnitude which will allow for reduced uncertainty in the source term. The room return shield will house and isolate the shielding samples, neutron activation foils, and other detectors. Data collected will provide a comparison of the neutron and gamma dose for shielded and unshielded configurations. This data will be used to validate shielding data used for modeling criticality accident alarm systems. Shielding samples included in this benchmark are C, Fe, NaCl, Pb, high density polyethylene, and SiO 2 . This paper details the design of the room return shield, the experimental plan, challenges, and mitigation strategies.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

NASA NTRSDate not supplied

Can We Model Forest Demography Globally? Benchmarking of State-of-the-Art Demographic DGVMs

Forests in Dynamic Global Vegetation Models (DGVMs) have historically been simulated as area-averaged plant functional types in each gridcell instead of representing communities of trees of different sizes and ages (demography). Just as the behaviour of a tree differs according to its ontogeny, so the behaviour of forests is known to differ depending on their demography. Accurately simulating demography is therefore key in order to address questions on afforestation and management strategies, as well as assessments of resilience of forests to disturbances such as drought and fire or diversity changes after a disturbance. Ultimately, demography determines the overall forest biomass in natural forests and is a key arbiter of growth and mortality rates. DGVMs are now able to simulate size and age structure of the trees in forests. However, these models have so far not been benchmarked alongside each other. We evaluate 6 DGVMs (BiomeE, CABLE-POP, FATES, LPJ-GUESS, JULES-RED, ORCHIDEE) against observations on regrowth dynamics as well as natural forests at boreal, temperate and tropical sites. We examine whether the models capture observed regrowth dynamics after disturbance, well-known stand size structure and well-established processes such as self-thinning. We outline the planned route forward towards a standardised international benchmarking framework for demographic DGVMs.

Dynamic Global Vegetation Models↗

DOE OSTI2022

HYSPLIT/MACCS Atmospheric Dispersion Model Technical Documentation and Benchmark Analysis

The nuclear accident consequence analysis code MACCS has traditionally modeled dispersion during downwind transport using a Gaussian plume segment model. MACCS is designed to estimate consequence measures such as air concentrations and ground depositions, radiological doses, and health and economic impacts on a statistical basis over the course of a year to produce annualaveraged output measures. The objective of this work is to supplement the Gaussian atmospheric transport and diffusion (ATD) model currently in MACCS with a new option using the HYSPLIT model. HYSPLIT/MACCS coupling has been implemented, with HYSPLIT as an alternative ATD option. The subsequent calculations in MACCS use the HYSPLIT-generated air concentration, and ground deposition values to calculate the same range of output quantities (dose, health effects, risks, etc.) that can be generated when using the MACCS Gaussian ATD model. Based on the results from the verification test cases, the implementation of the HYSPLIT/MACCS coupling is confirmed. This report contains technical details of the HYSPLIT/MACCS coupling and presents a benchmark analysis using the HYSPLIT/MACCS coupling system. The benchmark analysis, which involves running specific scenarios and sensitivity studies designed to examine how the results generated by the traditional MACCS Gaussian plume segment model compare to the new, higher fidelity HYSPLIT/MACCS modeling option, demonstrates the modeling results that can be obtained by using this new option. The comparisons provided herein can also help decision-makers evaluate the potential benefit of using results based on higher fidelity modeling with the additional computational burden needed to perform the calculations. Three sensitivity studies to investigate the potential impact of alternative modeling options, regarding 1) input meteorological data set, 2) method to estimate stability class, and 3) plume dispersion model for larger distances, on consequence results were also performed. The results of these analyses are provided and discussed in this report.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

Systematic benchmarking demonstrates large language models have not reached the diagnostic accuracy of traditional rare-disease decision support tools

Large language models (LLMs) show promise in supporting differential diagnosis, but their performance is challenging to evaluate due to the unstructured nature of their responses, and their accuracy compared to existing diagnostic tools is not well characterized. To assess the current capabilities of LLMs to diagnose genetic diseases, we benchmarked these models on 5213 previously published case reports using the Phenopacket Schema, the Human Phenotype Ontology and Mondo disease ontology. Prompts generated from each phenopacket were sent to seven LLMs, including four generalist models and three LLMs specialized for medical applications. The same phenopackets were used as input to a widely used diagnostic tool, Exomiser, in phenotype-only mode. The best LLM ranked the correct diagnosis first in 23.6% of cases, whereas Exomiser did so in 35.5% of cases. While the performance of LLMs for supporting differential diagnosis has been improving, it has not reached the level of commonly used traditional bioinformatics tools. Future research is needed to determine the best approach to incorporate LLMs into diagnostic pipelines.

Reese, Justin T. [Lawrence Berkeley National Labor↗

DOE OSTI2022

Impact of Increased Latent Generations on Sensitivity Calculations with SCALE

Analyses of cross section sensitivity data from systems with fissile material allow analysts to associate an importance for each material, nuclide, reaction, and neutron energy by simulating real world criticality scenarios. Although criticality safety validation efforts can be guided by the cross-section sensitivity and uncertainty data generated for a particular system, these calculations can often be computationally expensive and sometimes cumbersome without proper guidance. The TSUNAMI suite within the SCALE code package has several methods for generating sensitivity data, including multigroup and continuous energy (CE) capabilities. The release of SCALE 6.3 has three different CE methods for generating cross section sensitivity data: (1) the Iterated Fission Probability (IFP) method with the KENO Monte Carlo transport solver, (2) the IFP method with the Shift Monte Carlo transport solver, and (3) the Contributon-Linked eigenvalue sensitivity/Uncertainty estimation via Tracklength importance CHaracterization (CLUTCH) method with the KENO Monte Carlo transport solver. Although the CLUTCH method has additional parameters for generating sensitivity data files relative to the IFP method, all three methods use latent generations, which are the generations between an event (i.e., fission) and the assessment of importance based on the asymptotic population of progeny neutrons. Increasing the number of latent generations in a calculation leads to increased discrimination of the sensitivity coefficients but at the cost of the increased uncertainty associated with those generated values. Analysts must balance the accuracy of the sensitivity calculations and its uncertainty with the associated computational cost involved in generating the values. This paper discusses the impact of adjusting the latent generation parameter for a range of sensitivity values and how these changes compare with the direct perturbation values obtained from a change of ±0.5% Δ k in both benchmark and safety application models. Two benchmarks from the International Handbook of Evaluated Criticality Safety Benchmark Experiments and the MPC-32 dual purpose canister for spent nuclear fuel are used for analysis.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2023

U.S. Solar Photovoltaic System and Energy Storage Cost Benchmarks, With Minimum Sustainable Price Analysis: Q1 2023

The U.S. Department of Energy's (DOE's) Solar Energy Technologies Office (SETO) aims to accelerate the advancement and deployment of solar technology in support of an equitable transition to a decarbonized economy no later than 2050, starting with a decarbonized power sector by 2035. Its approach to achieving this goal includes driving innovations in technology, hardware, and soft cost reductions to make solar affordable and accessible for all. As part of this effort, SETO must track solar cost trends so it can focus its research and development (R&D) on the highest-impact activities. The benchmarks in this report are bottom-up cost estimates of all major inputs to PV and energy storage system installations. Bottom-up costs are based on national averages and do not necessarily represent typical costs in all local markets. Like last year's report, this year's report includes two distinct sets of benchmarks: minimum sustainable price (MSP) benchmarks and modeled market price (MMP) benchmarks. MSP benchmarks can be interpreted as the minimum price a company needs to charge to remain financially solvent in the long term based on the minimum sustainable prices of all inputs including minimum sustainable profit margins. MMP benchmarks can be interpreted as the actual cash sales price a company charges in the given benchmark period. These simplified estimates are useful for tracking technological progress, but they do not reflect all experiences. In fact, no individual estimate under any approach can reflect the diversity of the PV and storage manufacturing and installation industries. Our residential MMP benchmark ($2.90 per watt direct current [Wdc]) is 24% higher than the MSP benchmark ($2.34/Wdc) and 9% lower than our MMP benchmark ($3.18/Wdc) from Q1 2022 in 2022 U.S. dollars (USD). For community solar, our MMP benchmark ($1.75/Wdc) is 18% higher than our MSP benchmark ($1.49/Wdc). Our Q1 2022 benchmark report has no community solar system for comparison. For utility-scale systems with one-axis tracking, our MMP benchmark ($1.17/Wdc) is 22% higher than our MSP benchmark ($0.96/Wdc) and 10% higher than its counterpart ($1.07/Wdc) in Q1 2022 in 2022 USD.

14 SOLAR ENERGY↗

DOE OSTI2023

Q1 2023 U.S. Solar Photovoltaic System and Energy Storage Cost Benchmarks With Minimum Sustainable Price Analysis Data File

14 SOLAR ENERGY↗

NASA NTRS1997

BACT Simulation User Guide (Version 7.0)

This report documents the structure and operation of a simulation model of the Benchmark Active Control Technology (BACT) Wind-Tunnel Model. The BACT system was designed, built, and tested at NASA Langley Research Center as part of the Benchmark Models Program and was developed to perform wind-tunnel experiments to obtain benchmark quality data to validate computational fluid dynamics and computational aeroelasticity codes, to verify the accuracy of current aeroservoelasticity design and analysis tools, and to provide an active controls testbed for evaluating new and innovative control algorithms for flutter suppression and gust load alleviation. The BACT system has been especially valuable as a control system testbed.

Waszak, Martin R.↗

DOE OSTI2026

QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding

Quantum computing calibration depends on interpreting experimental data, and calibration plots provide the most universal human-readable representation for this task, yet no systematic evaluation exists of how well vision-language models (VLMs) interpret them. We introduce QCalEval, the first VLM benchmark for quantum calibration plots: 243 samples across 87 scenario types from 22 experiment families, spanning superconducting qubits and neutral atoms, evaluated on six question types in both zero-shot and in-context learning settings. The best general-purpose zero-shot model reaches a mean score of 72.3, and many open-weight models degrade under multi-image in-context learning, whereas frontier closed models improve substantially. A supervised fine-tuning ablation at the 9-billion-parameter scale shows that SFT improves zero-shot performance but cannot close the multimodal in-context learning gap. As a reference case study, we release NVIDIA Ising Calibration 1, an open-weight model based on Qwen3.5-35B-A3B that reaches 74.7 zero-shot average score.

Cao, Shuxiang↗

DOE OSTI2022

On the Fidelity of Computational Models for the Flow of Milled Loblolly Pine: A Benchmark Study on Continuum-Mechanics Models and Discrete-Particle Models

The upstream of bioenergy industry has suffered from unreliable operations of granular biomass feedstocks in handling equipment. Computational modeling, including continuum-mechanics models and discrete-particle models, offers insightful understandings and predictive capabilities on the flow of milled biomass and can assist equipment design and optimization. This paper presents a benchmark study on the fidelity of the continuum and discrete modeling approaches for predicting granular biomass flow. We first introduce the constitutive law of the continuum-mechanics model and the contact law of the coarse-grained discrete-particle model, with model parameters calibrated against laboratory characterization tests of the milled loblolly pine. Three classical granular material flow systems (i.e., a lab-scale rotating drum, a pilot-scale hopper, and a full-scale inclined plane) are then simulated using the two models with the same initial and boundary conditions as the physical experiments. The close agreement of the numerical predictions with the experimental measurements on the hopper mass flow rate, the hopper critical outlet width, the material stopping thickness on the inclined plane, and the dynamic angle of repose, clearly indicates that the two methods can capture the critical flow behavior of granular biomass. The qualitative comparison shows that the continuum-mechanics model outperforms in parameterization of materials and wall friction, and large-scale systems, while the discrete-particle model is more preferred for discontinuous flow systems at smaller scales. Industry stakeholders can use these findings as guidance for choosing appropriate numerical tools to model biomass material flow in part of the optimization of material handling equipment in biorefineries.

Jin, Wencheng↗

DOE OSTI2024

Two MCNP Models for Computational Performance Benchmarking

The purpose of this report is to describe two non-trivial MCNP models and execution configurations that are suitable to characterizing computational performance. To that end, this report uses a constructive solid geometry (CSG) representation of the Oak Ridge National Laboratory (ORNL) Pool Critical Assembly (PCA) [1–3] to perform a k-eigenvalue calculation and an unstructured mesh (UM) representation of the International Commission on Radiological Protection (ICRP) publication 145 (ICRP145) male human phantom [4, 5] to perform a fixed-source calculation assuming that a 1 MeV photon source is distributed throughout the phantom’s liver.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Downfolding from ab initio to interacting model Hamiltonians: comprehensive analysis and benchmarking of the DFT+cRPA approach

Abstract Model Hamiltonians are regularly derived from first principles to describe correlated matter. However, the standard methods for this contain a number of largely unexplored approximations. For a strongly correlated impurity model system, here we carefully compare a standard downfolding technique with the best possible ground-truth estimates for charge-neutral excited-state energies and wave functions using state-of-the-art first-principles many-body wave function approaches. To this end, we use the vanadocene molecule and analyze all downfolding aspects, including the Hamiltonian form, target basis, double-counting correction, and Coulomb interaction screening models. We find that the choice of target-space basis functions emerges as a key factor for the quality of the downfolded results, while orbital-dependent double-counting corrections diminish the quality. Background screening of the Coulomb interaction matrix elements primarily affects crystal-field excitations. Our benchmark uncovers the relative importance of each downfolding step and offers insights into the potential accuracy of minimal downfolded model Hamiltonians.

Chemistry↗