Search NASA⌕ Search

SEARCH · Search NASA

Results for “model benchmark”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

The NASA Ames Mars Global Climate Model: Benchmarking Publicly Released Source Code and Model Output

We have recently publicly released source code from the new NASA Ames Mars Global Climate Model (MGCM), which is based on NOAA/GFDL cubed-sphere finite volume (FV3-based) dynamical core (https://github.com/nasa/AmesGCM). We also we have a manuscript in preparation that aims to document the status of the new MGCM, and present selected simulations generated from it with interpretations and comparisons to both observations and the Ames Legacy MGCM. Output from our reference simulation will be made publicly available as well. One of our ongoing goals is to understand the underlying causes for differences between results produced with the new FV3-based dynamical core compared to the Legacy C-grid dynamical core. While the thermal and dynamical fields predicted with the new and Legacy GCMs are broadly similar for much of the year, there are key differences at low resolution when no external drag is applied to the new MGCM. This is particularly clear during a seasonal window of ~100 degrees of Ls surrounding southern summer solstice, when the predicted northern hemisphere polar warming is significantly over-predicted in the new MGCM. When we apply Rayleigh drag throughout much of the tropics and sub-tropics to the new MGCM, the simulated zonal mean structure of the atmosphere is much more consistent with both MCS observations and Legacy MGCM simulations. We note that the Kling et al. (2023; this meeting) study demonstrates that the behavior that we see with Rayleigh drag here can be recovered with high resolution simulations (in the horizontal and in the vertical) or with parameterized orographic and non-orographic gravity waves at lower resolution. While this work is still in progress, our preliminary conclusion is that the new dynamical core is less dissipative than the Legacy dynamical core. At low to moderate resolution, users of the new MGCM will need to be careful to use some sort of external drag, either in the form of gravity wave drag parameterizations or the simpler Rayleigh drag.

Melinda April Kahre↗
DOE OSTI2022

Benchmark Model for Wastewater Treatment Using an Activated Sludge Process

This is benchmark model for wastewater treatment using an activated sludge process. The activated sludge process is a means of treating both municipal and industrial wastewater. The activated sludge process is a multi-chamber reactor unit that uses highly concentrated microorganisms to degrade organics and remove nutrients from wastewater, producing quality effluent. This model provides pollutant concentrations, mass balance, electricity requirements, and treatment costs. This model will be continuously updated based on the latest data.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
NASA NTRSDate not supplied

Benchmark Models for Classification of Radiation Type Induced in Immune Cells

NASA Biological and Physical Sciences and the Science Mission Directorate have published a benchmark dataset of mouse immune cells subjected to radiation-induced DNA damage. The dataset comprises ML-ready microscopic imagery of said cells, including labels indicating radiation type and dose. The machine learning team at NASA Interagency Implementation and Advanced Concept Team (IMPACT) created multiple benchmark models. Initially, we conducted a preliminary analysis using thresholding. The algorithm used thresholds on average brightness of the available images to classify them into their respective radiation type. We also tested machine learning approaches. Convolutional Neural Networks (CNN) emerged as the best-performing model. This poster presents the benchmark scores obtained by the models.

Vishal Perekadan↗
DOE OSTI2022

Survey of Relevant Data from the MSRP to Guide Development of MSR Chemistry Modeling Benchmarks

The Multiphysics Applications technical area of the Nuclear Energy Advanced Modeling and Simulation (NEAMS) program is tasked with assessing, demonstrating, and applying NEAMS tools in solving multiphysics problems of nuclear reactors, such as liquid-fueled molten salt reactors (MSRs), which are the focus of this report. MSRs are actively being pursued as a potential candidate for power and heat generation by the nuclear industry. However, the inherent multiphysics nature of liquid-fueled MSRs, stemming from the strong interrelationship of neutronics, thermal fluids, and chemistry phenomena, provides unique challenges in modeling and simulation (mod/sim). Therefore, it will be important to develop mod/sim tools with varying types of multiphysics coupling that depend on the problem. The current work is focused on the initiation and development of MSR chemistry modeling benchmarks useful for validating current and potential future NEAMS tools. Development of such benchmarks include the following actions: 1) Summarize available chemistry data from the Oak Ridge National Laboratory (ORNL) MSR Program (MSRP) including operation of the Molten Salt Reactor Experiment (MSRE) and design studies for the Molten Salt Breeder Reactor (MSBR) concept; 2) Recommend simulation problems in MSR chemistry mod/sim based on (1); 3) Assess the current state of NEAMS tools that may support (2); 4) Demonstrate and validate the capabilities of the NEAMS tools in (3) while providing iterative feedback on future code development activities. The objective of this report is to initiate this effort by completing actions (1) and (2), which are discussed in Section 2. An overview of the relevant available MSRE experimental data is provided with examples of how this data may be useful in chemistry mod/sim problems, with the caveat that most of this data is over 50 years old, therefore some problem details as well as uncertainty estimates are often not provided. In Section 3, the current state of NEAMS tools is assessed for potential use in these mod/sim problems, with considerations for future code development activities, supporting action (3). Future work supporting this project under NEAMS may include a deeper dive into the specific phenomena discussed here including tasks such as the compilation of additional available data, updates in code development activities, and ultimately the demonstration and validation of these tools, supporting action (4).

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
NASA NTRS1993

Experimental unsteady pressures at flutter on the Supercritical Wing Benchmark Model

This paper describes selected results from the flutter testing of the Supercritical Wing (SW) model. This model is a rigid semispan wing having a rectangular planform and a supercritical airfoil shape. The model was flutter tested in the Langley Transonic Dynamics Tunnel (TDT) as part of the Benchmark Models Program, a multi-year wind tunnel activity currently being conducted by the Structural Dynamics Division of NASA Langley Research Center. The primary objective of this program is to assist in the development and evaluation of aeroelastic computational fluid dynamics codes. The SW is the second of a series of three similar models which are designed to be flutter tested in the TDT on a flexible mount known as the Pitch and Plunge Apparatus. Data sets acquired with these models, including simultaneous unsteady surface pressures and model response data, are meant to be used for correlation with analytical codes. Presented in this report are experimental flutter boundaries and corresponding steady and unsteady pressure distribution data acquired over two model chords located at the 60 and 95 percent span stations.

Dansberry, Bryan E.↗
DOE OSTI2020

Validation of MCNP Critical Benchmark Models of PU-MET-FAST-016 [Slides]

The validation of PU-MET-FAST-016 contributed to the centralized LANL benchmark repository currently under development. The revisions made the MCNP models statistically, significantly more similar to the benchmark models. The overall impact on the USL is negligible. The revisions to the PU-MET-FAST-016 models provide value to the Los Alamos Benchmark Suite without invalidating past Whisper results.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

AI Model Benchmarking for Nonproliferation Applications: Steel Thread Benchmarking Task Force Technical Report (Rev. 2)

Steel Thread is a NA-22 venture that seeks to build trustworthy, reliable AI models that can be used in a wide variety of nonproliferation tasks. A key aspect of building these models is developing appropriate benchmarks and evaluation methods, which will enable the venture to identify and adapt models to provide the most value in the nonproliferation domain. Benchmarks must be relevant to key tasks in this domain, such as question answering, information retrieval, document summarization and classification, consensus analysis, and image and data analysis. This report 1) provides an overview of benchmark design, evaluation, and challenges; 2) reviews a variety of open benchmarks, with a focus on language models and tasks; and 3) identifies benchmarks that are most relevant to Steel Thread. This report is intended to serve as a basis for further efforts to classify and evaluate benchmarks and their correlation with success on nonproliferation-specific tasks. The Steel Thread venture has defined benchmarks to be a particular combination of a dataset (or datasets) and a metric (or metrics) conceptualized as representing one or more specific tasks or sets of abilities for a specific modality. It is adopted by a research community as a shared framework for comparing methods.1 It includes 1) Data: Labeled (a designated subset not used for training, which could be all the data), 2) Metric: A way to quantify performance, 3) Task/Ability: What the benchmark is testing, 4) Protocol: A structured and repeatable evaluation process, 5) Baseline/Reference Model: For comparison; could be statistical, rule-based, SME-derived, or another model, and 6) Maintenance Plan: to update with new information over time; important for long-term utility. For further clarity, the definition includes what a benchmark, in this context, is not. It is not a corpus of training data, specific to a model (it is intended to apply to a range of models), a universal evaluation of performance, a guarantee that the ‘top’ model on the leaderboard will be the best fit for every specific use case, an all-encompassing proof of a model’s universal quality, nor is it a one-size-fits-all measure of success. It does not cover every real-world constraint (like operational, ethical, or cost considerations), a systems integration test, or a unit test. This definition was inspired by and resulted from discussions within the Steel Thread Benchmarking Task Force. This group was formed to define what we would mean as a benchmark within Steel Thread but persisted as the need to develop a thorough understanding of the large and expanding existing benchmarking space. This technical report is a result of the group’s divide and conquer approach to exploring this space. The release of benchmarks might not be progressing as quickly as model development, but it is moving very fast, as many benchmarks quickly become saturated, when state-of-the-art models score so close to the benchmark’s ceiling that their results are virtually indistinguishable. At that point, the test no longer differentiates between new systems, so researchers usually stop reporting scores as the benchmark no longer informs about improvements from the next generation of models. In the OpenAI announcement of GPT-5, they reported results on six flagship public benchmarks (AIME 2025, SWE-bench Verified, Aider Polyglot, MMMU, HealthBench Hard, GPQA) but the full system-card covers roughly thirty-five separate evaluations, comprising hundreds of test task items in total. There have been some efforts to summarize benchmarks in specific fields, like for text-to-image generation, but these surveys have had a narrow methodology scope. Therefore, a comprehensive survey of all benchmarks or even all benchmarks that could be relevant to Steel Thread is outside of the scope of this report. We chose some specific benchmarks to investigate in detail.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2016

2016 International Land Model Benchmarking (ILAMB) Workshop Report

As earth system models (ESMs) become increasingly complex, there is a growing need for comprehensive and multi-faceted evaluation of model projections. To advance understanding of terrestrial biogeochemical processes and their interactions with hydrology and climate under conditions of increasing atmospheric carbon dioxide, new analysis methods are required that use observations to constrain model predictions, inform model development, and identify needed measurements and field experiments. Better representations of biogeochemistryclimate feedbacks and ecosystem processes in these models are essential for reducing the acknowledged substantial uncertainties in 21st century climate change projections.

earth system models (ESMs)↗
NASA NTRS1992

Dynamic characteristics of a Benchmark models program supercritical wing

The dynamic characteristics of the Benchmark Supercritical Wing are presented. The Benchmark Supercritical Wing is a rigid semi-span model with a rectangular planform and a supercritical airfoil shape. This model will be tested on a flexible mount system that provides a well defined, two-degree-of-freedom dynamic system. This paper defines the dynamic characteristics of the model/mount system as determined in a preliminary ground vibration test. A general description of the model, flexible mount system, and instrumentation are included as well as a summary of the data set which the testing of this model is intended to produce.

Dansberry, Bryan E.↗
NASA NTRS2019

The Arctic-Boreal vulnerability experiment model benchmarking system

NASA's Arctic-Boreal Vulnerability Experiment (ABoVE) integrates field and airborne data into modeling and synthesis activities for understanding Arctic and Boreal ecosystem dynamics. The ABoVE Benchmarking System (ABS) is an operational software package to evaluate terrestrial biosphere models against key indicators of Arctic and Boreal ecosystem dynamics, i.e.: carbon biogeochemistry, vegetation, permafrost, hydrology, and disturbance. The ABS utilizes satellite remote sensing data, airborne data, and field data from ABoVE as well as collaborating research networks in the region, e.g.: the Permafrost Carbon Network, the International Soil Carbon Network, the Northern Circumpolar Soil Carbon Database, AmeriFlux sites, the Moderate Resolution Imaging Spectroradiometer, the Orbiting Carbon Observatory 2, and the Soil Moisture Active Passive mission. The ABS is designed to be interactive for researchers interested in having their models accurately represent observations of key Arctic indicators: a user submits model results to the system, the system evaluates the model results against a set of Arctic-Boreal benchmarks outlined in the ABoVE Concise Experiment Plan, and the user then receives a quantitative scoring of model strengths and deficiencies through a web interface. This interactivity allows model developers to iteratively improve their model for the Arctic-Boreal Region by evaluating results from successive model versions. We show here, for illustration, the improvement of the Lund–Potsdam–Jena-Wald Schnee und Landschaft (LPJwsl) version model through the ABoVE ABS as a new permafrost module is coupled to the existing model framework. The ABS will continue to incorporate new benchmarks that address indicators of Arctic-Boreal ecosystem dynamics as they become available.

ABoVE↗
DOE OSTI2025

Towards robust surrogate models: Benchmarking machine learning approaches to expediting phase field simulations of brittle fracture

Data-driven approaches have the potential to make modeling complex, nonlinear physical phenomena significantly more computationally tractable. For example, computational modeling of fracture is a core challenge where machine learning techniques have the potential to provide a much needed speedup that would enable progress in areas such as multi-scale modeling and uncertainty quantification. Currently, phase field modeling (PFM) of fracture is one such approach that offers a convenient variational formulation to model crack nucleation, branching and propagation. To date, machine learning techniques have shown promise in approximating PFM simulations. While standard fracture benchmarks represent realistic scenarios frequently observed in practice, they typically do not provide sufficiently challenging tests for data-driven methods. Here, to address this gap, we introduce a challenging dataset based on PFM simulations designed to benchmark and advance ML methods for fracture modeling. This dataset includes three energy decomposition methods, two boundary conditions, and 1000 random initial crack configurations for a total of 6000 simulations. Each sample contains 100 time steps capturing the temporal evolution of the crack field. Alongside this dataset, we also implement and evaluate Physics Informed Neural Networks (PINN), Fourier Neural Operators (FNO), and UNet models as baselines, and explore the impact of ensembling strategies on prediction accuracy. With this combination of our dataset and baseline models drawn from the literature we aim to provide a standardized and challenging benchmark for evaluating machine learning approaches to solid mechanics. Our results highlight both the promise and limitations of popular current models, and demonstrate the utility of this dataset as a testbed for advancing machine learning in fracture mechanics research.

Benchmark dataset↗
DOE OSTI2026

Rates of Sea‐Level Rise Are Highly Sensitive to Ice Viscosity Parameters in Model Benchmarks

Glacier flow plays a major role in current and future rates of globally averaged sea-level rise. The viscosity of glacial ice, controlling the rate of flow, decreases as stress increases and is highly sensitive to the value of the stress exponent, $n$, in the constitutive equation for viscous flow. Glaciologists and climate modelers almost exclusively assume $n=3$ when modeling ice flow and projecting sea-level rise through forward modeling. However, recent work suggests that $n\approx 4$ better fits observations, prompting the question: How sensitive are projections of sea-level rise to the value of $n$? We use an established community ice flow model and standard benchmark experiments designed as an idealized representation of Pine Island Glacier, West Antarctica. While initializing an $n=3$ model to match observations of an $n=4$ ice sheet is possible, we find that incorrectly assuming $n=3$ when in fact $n=4$ dramatically underestimates rates of sea-level rise. The scale of this error grows nonlinearly with the magnitude of the climate forcing, acting to increase projection uncertainties. Additionally, we find that models often account for this stress-dependent rheology mismatch during model initialization in a way that masks this rheological effect in the short term while leaving model outputs vulnerable to larger biases in longer-term projections. Initializations to observations of Pine Island Glacier display similar rheology-mismatch fingerprints to our idealized example.

climate sensitivity↗
NASA NTRS2016

Benchmarking Model Variants in Development of a Hardware-in-the-Loop Simulation System

Distributed engine control architecture presents a significant increase in complexity over traditional implementations when viewed from the perspective of system simulation and hardware design and test. Even if the overall function of the control scheme remains the same, the hardware implementation can have a significant effect on the overall system performance due to differences in the creation and flow of data between control elements. A Hardware-in-the-Loop (HIL) simulation system is under development at NASA Glenn Research Center that enables the exploration of these hardware dependent issues. The system is based on, but not limited to, the Commercial Modular Aero-Propulsion System Simulation 40k (C-MAPSS40k). This paper describes the step-by-step conversion from the self-contained baseline model to the hardware in the loop model, and the validation of each step. As the control model hardware fidelity was improved during HIL system development, benchmarking simulations were performed to verify that engine system performance characteristics remained the same. The results demonstrate the goal of the effort; the new HIL configurations have similar functionality and performance compared to the baseline C-MAPSS40k system.

Propulsion Simulation↗
NASA NTRS2016

Benchmarking Model Variants in Development of a Hardware-in-the-Loop Simulation System

Distributed engine control architecture presents a significant increase in complexity over traditional implementations when viewed from the perspective of system simulation and hardware design and test. Even if the overall function of the control scheme remains the same, the hardware implementation can have a significant effect on the overall system performance due to differences in the creation and flow of data between control elements. A Hardware-in-the-Loop (HIL) simulation system is under development at NASA Glenn Research Center that enables the exploration of these hardware dependent issues. The system is based on, but not limited to, the Commercial Modular Aero-Propulsion System Simulation 40k (C-MAPSS40k). This paper describes the step-by-step conversion from the self-contained baseline model to the hardware in the loop model, and the validation of each step. As the control model hardware fidelity was improved during HIL system development, benchmarking simulations were performed to verify that engine system performance characteristics remained the same. The results demonstrate the goal of the effort; the new HIL configurations have similar functionality and performance compared to the baseline C-MAPSS40k system.

Networked Systems↗
DOE OSTI2020

GoldSim E-Area Low-Level Waste Lacility Vadose Zone Model Benchmarking

This report documents the development and benchmarking of the E-Area Low-Level Waste Facility (ELLWF) GoldSim Engineered and Slit Trench (ET and ST) vadose zone models. Subsequent activities (beyond the scope of this effort) will couple the GoldSim-based vadose zone models to the GoldSim aquifer model, with the intent of performing stochastic analyses that couple all pertinent fate and transport processes from the ground surface up to the 100-m Point of Assessment (POA). The benchmarking presented within this report focuses on the Slit Trench 06 (ST06) bydrostratigraphic location that resides within the center set of slit trenches. These GoldSim-based vadose zone models are generic models capable of being benchmarked to any of the other trench locations within E-Area.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2022

Wind turbine response in waked inflow: A modelling benchmark against full-scale measurements

Predicting the power and loads of wind turbines in waked inflow conditions still presents a major modelling challenge. It requires the accurate modelling of the atmospheric flow conditions, wakes of upstream turbines and the response of the turbine of interest. Rigorous validations of model frameworks against measurements of utility-scale wind turbines in such scenarios remain limited to date. In this study, six models of different fidelity are compared against measurements from the DanAero experiment. The two benchmark cases feature a full-wake and partial-wake scenario, respectively. The simulations are compared against local pressure forces and inflow velocities measured on several blade sections of the downstream turbine, as well as met mast measurements and standard SCADA data. Regardless of the model fidelity, reasonable agreements are found in terms of the wake characteristics and turbine response. For instance, the azimuth variation of the mean aerodynamic forces acting on the blade was captured with a mean relative error of 15-20%. Additionally, while various model-specific deficiencies could be identified, the study highlights the need for further full-scale measurement campaigns with even more extensive instrumentation. Furthermore, it is concluded that validations should not be limited to integrated and/or time-averaged quantities that conceal characteristic spatial or temporal variations.

17 WIND ENERGY↗
DOE OSTI2022

Verification of MCNP Critical Benchmark Model of U233-COMP-THERM-004 [Slides]

The verification of U233-COMP-THERM-004 contributed to the LANL centralized repository that’s currently under development. Revisions made to the MCNP model were statistically significant. Magnitude of bias is relatively small in comparison to other uranium benchmark biases and thus will likely have not a significant effect on the USL.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗