Search NASA⌕ Search

SEARCH · Search NASA

Results for “model benchmark”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 415 records · Page 23
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

PickerXL, A Large Deep Learning Model to Measure Arrival Times from Noisy Seismic Signals

Precisely measuring seismic arrival times is a labor-intensive task but is critical for both earthquake monitoring and subsurface imaging. Recently published deep learning models have demonstrated superior performance compared to traditional automatic approaches for picking arrival times. Although existing deep learning models have shown promising results, further advancements are necessary as their performance is not yet satisfactory especially when applied to new regions and station networks. Increasing model size has led to improved performance in other machine learning applications. Here, we aimed to investigate whether enlarging deep learning models can increase performance on accepted benchmarks. We trained three models of varying sizes, small (1X), medium (4X), and large (16X), using globally distributed local and regional earthquake signals and background noise waveforms from a benchmark dataset, Stanford Earthquake Dataset. Our results indicate that the largest model (PickerXL) outperforms both the smaller models and Seisbench implementation of the PhaseNet model, which has the same number of parameters as our small model. The PickerXL model’s enhanced capacity to extract complex patterns from seismograms contributes to its superior arrival picking abilities compared to the smaller model.

Chai, Chengping [Oak Ridge National Laboratory (OR↗
DOE OSTI2025

A new metrics framework for quantifying and intercomparing atmospheric rivers in observations, reanalyses, and climate models

We present a new atmospheric river (AR) analysis and benchmarking tool, namely Atmospheric River Metrics Package (ARMP). It includes a suite of new AR metrics that are designed for quick analysis of AR characteristics via statistics in gridded climate datasets such as model output and reanalysis. This package can be used for climate model evaluation in comparison with reanalysis and observational products. Integrated metrics such as mean bias and spatial pattern correlation are efficient for diagnosing systematic AR biases in climate models. For example, the package identifies the fact that, in CMIP5 and CMIP6 (Coupled Model Intercomparison Project Phases 5 and 6) models, AR tracks in the South Atlantic are positioned farther poleward compared to ERA5 reanalysis, while in the South Pacific, tracks are generally biased towards the Equator. For the landfalling AR peak season, we find that most climate models simulate a completely opposite seasonal cycle over western Africa. This tool can also be used for identifying and characterizing structural differences among different AR detectors (ARDTs). For example, ARs detected with the Mundhenk algorithm exhibit systematically larger size, width, and length compared to the TempestExtremes (TE) method. The AR metrics developed from this work can be routinely applied for model benchmarking and during the development cycle to trace performance evolution across model versions or generations and set objective targets for the improvement of models. They can also be used by operational centers to perform near-real-time climate and extreme event impact assessments as part of their forecast cycle.

58 GEOSCIENCES↗
DOE OSTI2023

Search for dark matter produced in association with a Higgs boson decaying to tau leptons at $\sqrt{s}$ = 13 TeV with the ATLAS detector

A search for dark matter produced in association with a Higgs boson in final states with two hadronically decaying τ-leptons and missing transverse momentum is presented. The analysis uses 139 fb -1 of proton-proton collision data at $\sqrt{s}$ = 13 TeV collected by the ATLAS experiment at the Large Hadron Collider between 2015 and 2018. No evidence of physics beyond the Standard Model is found. The results are interpreted in terms of a 2HDM+a model featuring two scalar Higgs doublets and a pseudoscalar singlet field. Exclusion limits on the parameters of the model in selected benchmark scenarios are derived at 95% confidence level. Model-independent limits are also set on the visible cross-section for processes beyond the Standard Model producing missing transverse momentum in association with a Higgs boson decaying into τ-leptons.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2025

Exploring the hidden valley at MATHUSLA

Hidden valley models naturally predict numerous long-lived particles, the distinctive signatures of which would be compelling evidence for a hidden valley scenario. As these are typically low-energy particles, they pose a challenge in terms of passing energy triggers in traditional searches at the Large Hadron Collider. The MATHUSLA (MAssive Timing Hodoscope for Ultra-Stable neutraL pArticles) experiment is specifically designed for the purpose of detecting long-lived particles. It also has the capability of detecting lower energy particles relative to ATLAS and CMS. In this paper, we assess MATHUSLA's potential for effectively probing hidden valley models. As a benchmark, we assume the hidden valley sector communicates with Standard Model sectors via a heavy vector propagator that couples to Standard Model quarks as well as hidden valley quarks. We model the showering and hadronization in the hidden valley sector using PYTHIA and study the detector acceptance as a function of the hidden valley meson's lifetime. We find that MATHUSLA possesses significant capabilities to explore previously uncharted parameter space within hidden valley models.

Hidden sector↗
DOE OSTI2025

Benchmark study of a new simplified DFN model for shearing of intersecting fractures and faults

It is challenging to quantitatively predict shearing of intersecting fractures/faults because of dynamic frictional contacts accompanied by possible nonlinear rock deformation. To address such challenges, a new conceptual model—the simplified DFN model—was proposed and validated by Hu et al. 46 to use major paths (MPs) to represent complicated DFNs for calculation of shearing. In this work, we conducted a benchmark study for three examples that involve different levels of complexity of intersecting fractures, and correspondingly different numbers of MPs. The codes and software that were used in the benchmark cover a range of continuum, discontinuum and hybrid numerical methods: NMM (LBNL), FLAC3D (LBNL), GBDEM (KIGAM), FRACOD (DynaFrax), and CASRock (CAS). The general consistency between DFN and MP cases as predicted by all the codes/software demonstrates that major paths can be used to simplify the geometry of DFNs in a wide range of software. Disagreement in results made by some software and potential future improvements are discussed. We show that (1) shearing of one or multiple major fractures can be reduced if there are multiple smaller intersecting fractures in that area, which is a useful basis for understanding and controlling induced seismicity and merits further analysis, and (2) the agreement achieved in the benchmark examples provide confidence that the simplified DFN model is a promising conceptual model that can be used for different types of numerical approaches and software for simplifying the analysis of the shearing of intersecting fractures and faults.

58 GEOSCIENCES↗
DOE OSTI2025

AGR 1 and 2 Isotopic Depletion Validation

This presentation includes work performed developing a computational benchmark for the AGR 1 and 2 experiments. These experiments involve irradiation of TRISO particle compact in the Advanced Test Reactor (ATR). Many post-irradiation measurements were made from these experiments, including isotopic concentrations of fission products. The benchmark involves a MCNP-ORIGEN coupled model of the ATR and AGR experiment capsules to calculate a comprehensive inventory of fission product and actinide concentrations in the compacts. This presentation describes the experiments, the measurements obtained, the benchmark model, and results of the model evaluation.

97 - MATHEMATICS AND COMPUTING↗
DOE OSTI2026

PowerModelsGAT-AI: Physics-Informed Graph Attention for Multi-System Power Flow With Continual Learning

Solving the alternating current power flow equations in real time is essential for secure grid operation, yet classical Newton–Raphson solvers can be slow under stressed conditions. Existing graph neural networks for power flow are typically trained on a single system and often degrade on different systems. We present PowerModelsGAT-AI, a physics-informed graph attention network that predicts bus voltages and generator injections. The model uses bus-type-aware masking to handle different bus types and balances multiple loss terms, including a power-mismatch penalty, using learned weights. We evaluate the model on 14 benchmark systems (4 to 6,470 buses) and train a unified model on 13 of these under contingency conditions with up to two branch outages, achieving an average normalized mean absolute error of 0.89% for voltage magnitudes and R 2 >0.99 for voltage angles. We also show continual learning: when adapting a base model to a new 1,354-bus system, standard fine-tuning causes severe forgetting with error increases exceeding 1000% on base systems, while our experience replay and elastic weight consolidation strategy keeps error increases below 2% and in some cases improves base-system performance. Interpretability analysis shows that learned attention weights correlate with physical branch parameters (susceptance: r=0.38 ; thermal limits: r=0.22 ), and feature importance analysis supports that the model captures established power flow relationships.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

Deforestation reshapes land-surface energy-flux partitioning

Land-use and land-cover change significantly modify local land-surface characteristics and water/energy exchanges, which can lead to atmospheric circulation and regional climate changes. In particular, deforestation accounts for a large portion of global land-use changes, which transforms forests into other land cover types, such as croplands and grazing lands. Many previous efforts have focused on observing and modeling land–atmosphere–water/energy fluxes to investigate land–atmosphere coupling induced by deforestation. However, interpreting land–atmosphere–water/energy-flux responses to deforestation is often complicated by the concurrent impacts from shifts in land-surface properties versus background atmospheric forcings. In this study, we used 29 paired FLUXNET sites, to improve understanding of how deforested land surfaces drive changes in surface-energy-flux partitioning. Each paired sites included an intact forested and non-forested site that had similar background climate. We employed transfer entropy, a method based on information theory, to diagnose directional controls between coupling variables, and identify nonlinear cause–effect relationships. Transfer entropy is a powerful tool to detective causal relationships in nonlinear and asynchronous systems. The paired eddy covariance flux measurements showed consistent and strong information flows from vegetation activity (gross primary productivity (GPP)) and physical climate (e.g. shortwave radiation, air temperature) to evaporative fraction (EF) over both non-forested and forested land surfaces. More importantly, the information transfers from radiation, precipitation, and GPP to EF were significantly reduced at non-forested sites, compared to forested sites. We then applied these observationally constrained metrics as benchmarks to evaluate the Energy Exascale Earth System Model (E3SM) land model (ELM). ELM predicted vegetation controls on EF relatively well, but underpredicted climate factors on EF, indicating model deficiencies in describing the relationships between atmospheric state and surface fluxes. Moreover, changes in controls on surface energy flux partitioning due to deforestation were not detected in the model. We highlight the need for benchmarking model simulated surface-energy fluxes and the corresponding causal relationships against those of observations, to improve our understanding of model predictability on how deforestation reshapes land surface energy fluxes.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Demography, dynamics and data: building confidence for simulating changes in the world's forests

Vegetation demographic models (VDMs) are advanced tools for simulating forest responses to climate and land-use changes, and are essential for projecting carbon cycling and large-scale forest management strategies. Despite their increasing incorporation into Earth System Models, VDMs differ in their demographic assumptions, with no prior quantitative comparison of their performance. We benchmarked nine VDMs against observational data from boreal, temperate and tropical sites, assessing their accuracy in predicting tree growth, carbon turnover, biomass stocks and size distributions. Models were simulated under consistent climate conditions with postdisturbance recovery monitored for at least 420 yr. Postdisturbance carbon recovery trajectories showed significant variability while remaining within observational ranges. Initial regrowth rates varied substantially (0.03-0.60, 0.18-0.70 and 0.35-1.10 kgCm-2 yr-1 for boreal, temperate and tropical sites, respectively), influenced by each model's initial forest state. Models captured mature forest carbon content but showed compensating effects between overestimated growth and underestimated mortality rates. This first multi-model benchmarking identifies growth and mortality rates as critical calibration targets and highlights the need to refine postdisturbance establishment conditions for model development. We outline specific benchmarking variables needed to improve predictions of forest responses to environmental change.

demographic vegetation model benchmarking↗
DOE OSTI2021

Utilizing ensemble learning for performance and power modeling and improvement of parallel cancer deep learning CANDLE benchmarks

Abstract Machine learning (ML) continues to grow in importance across nearly all domains in modeling to learn from data. Often a tradeoff exists between a model's ability to minimize bias and variance. In this article, we utilize ensemble learning to combine linear, nonlinear, and tree‐/rule‐based ML methods to cope with the bias‐variance tradeoff and result in more accurate models. We use the datasets collected for two parallel cancer deep learning CANDLE benchmarks, NT3 and P1B2, to build performance and power models based on hardware performance counters using single‐object and multiple‐objects ensemble learning to identify the most important counters for improvement on the Cray XC40 Theta at Argonne National Laboratory. Based on the insights from these models, we improve the performance and energy of P1B2 and NT3 by optimizing the deep learning environments TensorFlow, Keras, Horovod, and Python under the huge page size of 8 MB. Experimental results show that ensemble learning not only produces more accurate models but also provides more robust performance counter ranking. We achieve up to 61.15% performance improvement and up to 62.58% energy saving for P1B2 and up to 55.81% performance improvement and up to 52.60% energy saving for NT3 on up to 24,576 cores.

Wu, Xingfu↗
DOE OSTI2020

Machine Learning for Materials Scientists: An Introductory Guide toward Best Practices

This Methods/Protocols article is intended for materials scientists interested in performing machine learning-centered research. Herein, we cover broad guidelines and best practices regarding the obtaining and treatment of data, feature engineering, model training, validation, evaluation and comparison, popular repositories for materials data and benchmarking data sets, model and architecture sharing, and finally publication. In addition, we include interactive Jupyter notebooks with example Python code to demonstrate some of the concepts, workflows, and best practices discussed. Overall, the data-driven methods and machine learning workflows and considerations are presented in a simple way, allowing interested readers to more intelligently guide their machine learning research using the suggested references, best practices, and their own materials domain expertise.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Benchmarking Simulated Precipitation in Earth System Models

Earth system models (ESMs) bridge observationally based and theoretical understanding of the Earth system. They are among the most often used tools to study a variety of questions related to variability and changes in Earth’s climate. For many applications, ESMs must realistically simulate observed large-scale precipitation patterns and seasonal cycles that have a multitude of societal and national security implications. Despite steady improvement in the simulation of precipitation, model errors in many aspects of precipitation characteristics limit the use of ESMs both in understanding Earth system variability and change and for decision-making.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Instrumentation for the In-Core Real-Time Mechanical Testing of Structural Materials (INCREASE) Project

Idaho National Laboratory (INL), in collaboration with the Electric Power Research Institute (EPRI), the Nuclear Regulatory Commission (NRC), the French Atomic and Alternative Energies Commission (CEA), the Joint Research Center (JRC), the Nuclear Research and Consultancy Group (NRG), and the Research Center Rez (CVR), started a Joint Experimental Program (JEEP) project that operates within the Nuclear Energy Agency’s Framework for Irradiation Experiments (FIDES II) program in order to develop capabilities for the in-core real-time mechanical testing of structural materials. This effort will focus on designing a shared capsule capable of housing a variety of in-core mechanical testing instrumentation allowing enhanced experiments for the material science community. The outcome of the project would be high-priority, stress relaxation data for stainless-steel-based materials provided by EPRI and CEA. Stress relaxation is a major phenomenon that contributes to material degradation in nuclear reactor components. Currently, nuclear material stress relaxation is assessed both before and after irradiation, using complex and costly post-irradiation examination (PIE) activities. In-situ data would support the development of precision modeling and simulation of this degradation phenomena and would provide validation and benchmarking for existing models using the PIE data. As part of the U.S. Department of Energy Advanced Sensor and Instrumentation (ASI) program, INL has fabricated and tested out-of-core mechanical test instrumentation. This instrumentation was designed for easy adaptation to the irradiation capsule proposed under this JEEP, and can be deployed to measure real-time stress relaxation under pressurized-water reactor (PWR) conditions. This initial effort will partially serve to replace the testing capabilities lost because of shutting down the Halden Boiling Water Reactor (HBWR). The project would provide these capabilities to the international community via a shared capsule design that is easily adaptable to additional material test reactors. The design features will incorporate expansion to PWR and non-light-water reactor (LWR) environments that will be developed in future work. The capsule and instrumentation will be demonstrated in the Massachusetts Institute of Technology Reactor (MITR) for phase I and the Petten High Flux Reactor (HFR) for phase II irradiations to deliver real-time stress relaxation data on high priority stainless steel structural materials.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Controls and relationships of soil organic carbon abundance and persistence vary across pedo–climatic regions

One of the largest uncertainties in the terrestrial carbon cycle is the timing and magnitude of soil organic carbon (SOC) response to climate and vegetation change. This uncertainty prevents models from adequately capturing SOC dynamics and challenges the assessment of management and climate change effects on soils. Reducing these uncertainties requires simultaneous investigation of factors controlling the amount (SOC abundance) and duration (SOC persistence) of stored C. We present a global synthesis of SOC and radiocarbon profiles (n Profile = 597) to assess the timescales of SOC storage. We use a combination of statistical and depth–resolved compartment models to explore key factors controlling the relationships between SOC abundance and persistence across pedo–climatic regions and with soil depth. This allows us to better understand (i) how SOC abundance and persistence covary across pedo–climatic regions and (ii) how the depth dependence of SOC dynamics relates to climatic and mineralogical controls on SOC abundance and persistence. We show that SOC abundance and persistence are differently related; the controls on these relationships differ substantially between major pedo–climatic regions and soil depth. For example, large amounts of persistent SOC can reflect climatic constraints on soils (e.g., in tundra/polar regions) or mineral absorption, reflected in slower decomposition and vertical transport rates. In contrast, lower SOC abundance can be found with lower SOC persistence (e.g., in highly weathered tropical soils) or higher SOC persistence (e.g., in drier and less productive regions). We relate variable patterns of SOC abundance and persistence to differences in the processes constraining plant C input, microbial decomposition, vertical C transport and mineral SOC stabilization potential. This process–oriented grouping of SOC abundance and persistence provides a valuable benchmark for global C models, highlighting that pedo–climatic boundary conditions are crucial for predicting the effects of climate change and soil management on future C abundance and persistence.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Economic Model for Estimation of GDP Losses in the MACCS Offsite Consequence Analysis Code

The MACCS (MELCOR Accident Consequence Code System) code is the U.S. Nuclear Regulatory Commission (NRC) tool used to perform probabilistic health and economic consequence assessments for atmospheric releases of radionuclides. It is also used by international organizations, both reactor owners and regulators. It is intended and most commonly used for hypothetical accidents that could potentially occur in the future rather than to evaluate past accidents or to provide emergency response during an ongoing accident. It is designed to support probabilistic risk and consequence analyses and is used by the NRC, U.S. nuclear licensees, the Department of Energy, and international vendors, licensees, and regulators. This report describes the modeling framework, implementation, verification, and benchmarking of a GDP-based model for economic losses that has recently been developed as an alternative to the original cost-based economic loss model in MACCS. The GDP-based model has its roots in a code developed by Sandia National Laboratories for the Department of Homeland Security to estimate short-term losses from natural and manmade accidents, called the Regional Economic Accounting analysis tool (REAcct). This model was adapted and modified for MACCS and is now called the Regional Disruption Economic Impact Model (RDEIM). It is based on input-output theory, which is widely used in economic modeling. It accounts for direct losses to a disrupted region affected by an accident, indirect losses to the national economy due to disruption of the supply chain, and induced losses from reduced spending by displaced workers. RDEIM differs from REAcct in its treatment and estimation of indirect loss multipliers, elimination of double counting associated with inter-industry trade in the affected area, and that it is designed to be used to estimate impacts for extended periods that can occur from a major nuclear reactor accident, such as the one that occurred at the Fukushima Daiichi site in Japan. Most input-output models do not account for economic adaptation and recovery, and in this regard RDEIM differs from its parent, REAcct, because it allows for a user-definable national recovery period. Implementation of a recovery period was one of several recommendations made by an independent peer review panel to ensure that RDEIM is state-of-practice. For this and several other reasons, RDEIM differs from REAcct. Both the original and the RDEIM economic loss models account for costs from evacuation and relocation, decontamination, depreciation, and condemnation. Where the original model accounts for an expected rate of return, based on the value of property, that is lost during interdiction, the RDEIM model instead accounts for losses of GDP based on the industrial sectors located within a county. The original model includes costs for disposal of crops and milk that the RDEIM model currently does not, but these costs tend to contribute insignificantly to the overall losses. This document discusses three verification exercises to demonstrate that the RDEIM model is implemented correctly in MACCS. It also describes a benchmark study at five nuclear power plants chosen to represent the spectrum of U.S. commercial sites. The benchmarks provide perspective on the expected differences between the RDEIM and the original cost-based economic loss models. The RDEIM model is shown to consistently predict larger losses than the original model, probably in part because it accounts for national losses by including indirect and induced losses; whereas, the original model only accounts for regional losses. Nonetheless, the RDEIM model predicts losses that are remarkably consistent with the original cost-based model, differing by 16% at most for the five sites combined with three source terms considered in this benchmark.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2026

Benchmarking of three DWM-based wake models at below-rated wind speeds

Wind turbine wake models are essential tools for predicting power losses and structural loads in wind farms. Among these, the dynamic wake meandering (DWM) model, included as a recommended approach in the International Electrotechnical Commission design standard, is a widely used engineering-fidelity method that balances accuracy and computational cost. This study compares the performance of three DWM-based wake model implementations (from the Technical University of Denmark, the National Renewable Energy Laboratory, and the Institute for Energy Technology) under below-rated wind speed conditions. Model predictions of wake flow, power output, and structural loads for a four-turbine row are evaluated across different ambient turbulence levels and wind-direction misalignments and compared against high-fidelity large-eddy simulation results. All three models captured the overall wake evolution and mean turbine performance with reasonable accuracy; their predicted time-averaged thrust and power were typically within 5 %–10 % of the large-eddy simulation benchmark. However, notable differences emerged in wake structure and unsteady load predictions, with discrepancies increasing for turbines further downstream. These differences highlight the importance of modelling choices such as wake summation and turbulence treatment, which strongly influence power-deficit and fatigue-load predictions. Comparison with large-eddy simulations reveals each approach's strengths and weaknesses, indicating where improvements are needed. Overall, the findings point to specific refinements for DWM models to improve their fidelity, ultimately enabling more robust wake predictions for wind farm design and operation.

17 WIND ENERGY↗