Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmark Tests”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 577 records · Page 32
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

BigNeuron: a resource to benchmark and predict performance of algorithms for automated tracing of neurons in light microscopy datasets

BigNeuron is an open community bench-testing platform with the goal of setting open standards for accurate and fast automatic neuron tracing. We gathered a diverse set of image volumes across several species that is representative of the data obtained in many neuroscience laboratories interested in neuron tracing. Here, we report generated gold standard manual annotations for a subset of the available imaging datasets and quantified tracing quality for 35 automatic tracing algorithms. The goal of generating such a hand-curated diverse dataset is to advance the development of tracing algorithms and enable generalizable benchmarking. Together with image quality features, we pooled the data in an interactive web application that enables users and developers to perform principal component analysis, t-distributed stochastic neighbor embedding, correlation and clustering, visualization of imaging and tracing data, and benchmarking of automatic tracing algorithms in user-defined data subsets. The image quality metrics explain most of the variance in the data, followed by neuromorphological features related to neuron size. Furthermore, we observed that diverse algorithms can provide complementary information to obtain accurate results and developed a method to iteratively combine methods and generate consensus reconstructions. The consensus trees obtained provide estimates of the neuron structure ground truth that typically outperform single algorithms in noisy datasets. However, specific algorithms may outperform the consensus tree strategy in specific imaging conditions. Finally, to aid users in predicting the most accurate automatic tracing results without manual annotations for comparison, we used support vector machine regression to predict reconstruction quality given an image volume and a set of automatic tracings.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

A curated benchmark for cofolding models on kinase conformational states

Abstract Protein kinases are critical drug targets, requiring therapeutics that can modulate their active and inactive conformational states. While cofolding models can generate global folds directly from kinase sequences and ligand SMILES strings, these models have not yet been tested on their ability to recover ligand-induced-fit conformational states of the kinase proteins. Here, we introduce KinConfBench, a curated benchmark of 2225 high-quality human kinase chains to evaluate the ability of four state-of-the-art cofolding models—Boltz-2, Chai-1, Protenix, and RoseTTAFold-All-Atom—to recover both canonical and rare conformational states. We show that geometric success metrics of a ligand pose in the active site do not correlate strongly with the correct kinase conformational state, motivating a new set of dynamical benchmarks for assessing cofolding models. While all four cofolding models achieve ~60–80% prediction accuracy for kinase conformational classification, they exhibit severe mode collapse when performing multiple inferences, show negligible structural diversity in sampling induced-fit motions, and display a prevalent “apo-drift” in which most cofolding models predominantly predict the kinase to be in its ligand-free state. Our results highlight that capturing ligand-induced protein conformational diversity, not just geometric fit, is critical for next-generation structure-based drug discovery.

Sun, Kunyang↗
DOE OSTI2023

MUSiC: Critical Experiment with Bare Highly Enriched Uranium Shells Benchmark

The measurements of the Uranium Subcritical and Critical (MUSiC) experiment were performed from December 2020 through April 2021. The purpose of the experiment was to test a variety of multiplicity detection systems for a large range of multiplication values and to see how well they perform as the k eff of the MUSiC series of experiments approached the delayed critical window. Out of the ten configurations that were studied, two of them were super critical and were recently approved as benchmarks and they will be included in the International Criticality Safety Benchmark Evaluation Project (ICBESP) handbook. For each of the two experiments, uncertainties were evaluated six categories: (1) critical measurement, (2) mass, (3) composition, (4) positioning, (5) dimensions, and (6) temperature. The largest contribution to the overall uncertainty in each of the cases was due to the uncertainties in the dimensions of the HEU Rocky Flats shells. Simplified detailed models were created with an average bias of -130.5 pcm due to the removal various components surrounding the experiment. For MCNP6.2 models using ENDF-VIII.0 neutron cross section data, the average difference between the calculated and benchmark k effs was -54 pcm for the detailed model and -102 pcm for the simplified model.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2024

Quantum-Hardware Focused Application Performance Benchmarks (Final Technical Report)

Quantum computers promise to transform how we do scientific calculations. In this project, we benchmark different current quantum computers for solving chemistry problems and find ways to build noise-resilient implementations. We also use the characterization techniques developed to test ion trap quantum computers.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2026

Drive Cycles, Battery Pack Scaling, and Usage Considerations for Long-Haul and Regional-Haul Electric Trucks

Electrifying Class-8 heavy-duty trucks presents a promising opportunity to enhance energy efficiency and reduce freight transport costs. Battery electric trucks (BETs), once considered niche, are gaining traction due to advancements in battery technology and cost reductions. However, accurately predicting battery lifespan under realistic usage conditions remains a key challenge. Understanding battery failure mechanisms and their links to design, operation, and management is essential for developers and fleet operators. This study introduces a method to develop simplified, lab-testable dynamic stress test (DST) cycles for regional and long-haul Class-8 BETs, derived from real-world diesel truck usage. These DSTs enable benchmarking of battery technologies, identification of aging stressors, and optimization of battery design, life, and cost. The approach supports evaluation of key metrics such as levelized cost of driving and total cost of ownership, aiding fair comparisons and adoption decisions. We also propose feasible battery pack sizes that meet current driving demands with strategic charging, and a method to scale pack-level DSTs to cell-level cycles for lab-based testing. These tools facilitate tradeoff analysis across battery chemistries, pack sizing, and charging strategies, while offering means to get insights into battery aging under realistic conditions-ultimately supporting informed BET deployment decisions.

25 ENERGY STORAGE↗
DOE OSTI2021

Benchmark estimate of the effect of anthropogenic emissions on the ocean surface

Abstract Investigations into the role of anthropogenic emissions in the occurrence of extreme weather often use a method that compares simulations of atmospheric climate models run under a factual scenario of historical boundary conditions observed during the period of the event against simulations run under a counterfactual scenario of what those boundary conditions might naturally have been over that same period in the absence of anthropogenic emissions. A particular requirement for this experiment design is an accurate estimation of ocean surface boundary conditions for use by the counterfactual natural simulations. Here we use output from the CMIP5 multi‐climate‐model archive to develop a robust estimate of sea surface temperatures and sea ice conditions for use in counterfactual natural simulations, intended as a benchmark estimate to facilitate comparison across climate models and across studies. This development includes tests to ensure that the final estimate is stable from year‐to‐year and stable against other perturbations to the methodology, as well as consideration of the strengths and weaknesses in comparison to other available attributable warming estimates. While this estimate is tailored specifically for the International CLIVAR C20C+ Detection and Attribution Project, it can be used by related projects as well.

Stone, Dáithí A.↗
DOE OSTI2026

Reactive Transport Modeling with Physics-Informed Machine Learning for Critical Minerals Applications

This study presents a physics-informed neural network (PINN) framework for reactive transport modeling for simulating fast bimolecular reactions in porous media. Accurate characterization of cAhemical interactions and product formation in surface and subsurface environments is essential for advancing critical mineral extraction and related geoscience applications. The proposed methodology sequentially addresses the flow and diffusion–reaction subproblems. The flow field is computed using a mixed formulation, while the diffusion–reaction system is modeled via two uncoupled tensorial diffusion equations reformulated in terms of chemical invariants. PINNs are employed to solve the governing equations, enabling data-efficient, mesh-free prediction of chemical concentration fields. The framework is validated through a series of benchmark problems involving flow in heterogeneous porous media. Initial verification is conducted using patch tests for the flow field, followed by validation of the transport problem with emphasis on preserving non-negativity of concentrations. The complete fast bimolecular reaction scenario is then solved, yielding spatial distributions of reactants and product species. Results demonstrate that the PINNs-based approach effectively captures sharp, mixing-limited reaction fronts and dispersive mixing behavior, offering reliable predictions of reactive plume evolution. These capabilities are crucial for evaluating long-term subsurface behavior in applications such as fluid storage, energy extraction, and efficient extraction of critical minerals.

42 ENGINEERING↗
DOE OSTI2026

Unimolecular Dynamics of Partially Deuterated Hydroperoxyalkyl Intermediates (•QOOD) in Cyclohexane and Cyclopentane Oxidation

The hydroperoxyalkyl radicals (•QOOH) formed in cyclohexane and cyclopentane oxidation with carbon radical center at the β site relative to the hydroperoxy group have recently been observed through their infrared fingerprint and time- and energy-resolved unimolecular dissociation dynamics to hydroxyl (OH) radical and cyclic ether products. Partial deuteration shifts the IR transitions associated with OH/OD motion to lower frequencies, providing a new window on the IR spectroscopy and unimolecular dynamics of these transient intermediates. Specifically, the overtone OD stretch (2ν OD ) has been identified at 5215.0 cm –1 and 5216.0 cm –1 for the β-QOOD intermediates in cyclohexane and cyclopentane oxidation, respectively. Unimolecular decay rates of the •QOOD intermediates at these energies have been obtained from the time-resolved appearance of OD products. The experimental rates are compared with statistical microcanonical rates evaluated using RRKM theory, including heavy-atom tunneling associated with simultaneous O–O bond elongation and C–O–O angle contraction along the reaction pathway. The experimental rates provide a further test of the previously determined transition state (TS) barriers, which were computed utilizing a benchmarking approach that builds on higher-level reference calculations for the smaller ethane oxidation system. Here, the experimental rate measurements for both •QOOD intermediates agree well with the computed rates after accounting for small changes in zero-point energy and a minor empirical adjustment of the TS barriers in both systems.

Ethers↗
DOE OSTI2025

Efficient Training of Deep Neural Operator Networks via Randomized Sampling

Neural operators (NOs) employ deep neural networks to learn the mappings between infinitedimensional function spaces. Deep operator network (DeepONet), a popular NO architecture, has demonstrated success in the real-time prediction of complex dynamics across various scientific and engineering applications. In this work, we introduce a random sampling technique to be adopted during the training of DeepONet, aimed at improving the generalization ability of the model, while significantly reducing the computational time. The proposed approach targets the trunk network of the DeepONet model that outputs the basis functions corresponding to the spatiotemporal locations of the bounded domain on which the physical system is defined. While constructing the loss function, DeepONet training traditionally considers a uniform grid of spatiotemporal points at which all the output functions are evaluated for each iteration. This approach leads to a larger batch size, resulting in poor generalization and increased memory demands, due to the limitations of the stochastic gradient descent (SGD) optimizer. The proposed random sampling over the inputs of the trunk net mitigates these challenges, improving generalization and reducing the memory requirements during training, resulting in significant computational gains. We validate our hypothesis through three benchmark examples, demonstrating substantial reductions in training time while achieving comparable or lower overall test errors relative to the traditional training approach. Our results indicate that incorporating randomization in the trunk network inputs during training enhances the efficiency and robustness of DeepONet, offering a promising avenue for improving the framework’s performance in modeling complex physical systems.

Karumuri, Sharmila [Department of Civil & Systems ↗
DOE OSTI2021

Cumulative $\chi^2$ Metric for VALID for ENDF/B-VII.1 and ENDF/B-VIII.0 in SCALE 6.3b9

Nuclear data testing is generally performed in a manner similar to validation: a series of critical benchmark experiments is modeled, and the differences between calculated and expected results are examined to understand the performance of a code and cross section set. This process is repeated for multiple sets of experiments with different transport codes and sometimes with different datasets. Occasionally, data processing differences are also important in understanding the differences in calculated results. Relevant experts then meet periodically to compare results and set priorities for improvement in the next round of data updates. This process has been largely effective, despite the confounding factors of different benchmark sets, application areas, sponsors, nationalities, and the resulting different priorities.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

High-Temperature Gas-Cooled Reactors Multiphysics Simulation Demonstration and Code Validation

This study presents a comprehensive benchmarking and verification effort of several thermal-hydraulic and multiphysics capabilities for high-temperature gas-cooled reactor applications. The first part of this effort focuses on the running-in verification of Griffin’s multiphysics capabilities, specifically for simulating the evolution of pebble-bed reactor cores from startup to equilibrium. Since Fiscal Year 2024, improvements and enhancements have been implemented in Griffin, including simplifying the process to specify streamlines and developing the online cross-section generation capability. In the absence of validation data, code-to-code comparisons are conducted with kugelpy, showing good agreement for integral quantities like k-eff predictions and predictions for maximum power density. However, accuracy issues are noted for more detailed quantities like the spatial distribution of fission rate densities which will require further work to address. The second part of this report presents an improved System Analysis Module (SAM) core channel model where the effects of cross flow are considered during the pressurized loss of forced cooling transient, resulting in an improved agreement of the predicted pebble temperature with respect to the predictions from the SAM 2D porous media model. Additionally, the wall channeling effect due to variable porosity at the near wall region of the core is also investigated. Furthermore, to demonstrate Griffin’s online cross-section generation capability, a Multiphysics simulation is performed by coupling Griffin to the SAM core channel model. In the third part of the report, as a part of the Organisation for Economic Co-operation and Development/Nuclear Energy Agency (OECD/NEA) thermal-hydraulic code validation benchmark activity for a high-temperature gas-cooled reactor, the High Temperature Test Facility (HTTF) is investigated first using the NekRS computational fluid dynamics (CFD) code to study the flow mixing phenomenon in the lower plenum of the facility. Then, code-to-code and code-to-data comparisons are performed for Test PG27, which is a pressurized conduction cooldown (PCC) test, using five different codes by six organizations from five countries. The different simulations show good agreements in terms of the general trend but there are differences in some results such as the peak temperatures of different regions and heat removal rate.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2025

Improvement and Verification of Online Cross Section Generation Capability of Griffin for TRISO-fueled Reactors

Griffin, a MOOSE-based reactor multiphysics code jointly developed by Idaho National Laboratory and Argonne National Laboratory under the DOE Office of Nuclear Energy’s NEAMS program, has pursued the development of an online multigroup cross section generation capability for a few years to enable high-fidelity, problem-dependent neutronics analyses of advanced thermal reactors. Recent advancements in Griffin’s online multigroup cross section generation capability have significantly improved the accuracy, robustness, and efficiency of self-shielding calculations for both prismatic and pebble-bed TRISO-fueled reactor applications. Key developments include a unified fuel self-shielding method applicable to both TRISO and annular compact/spherical shell fuel zone geometries; an advanced Dancoff Category-based Equivalence Theory using a bell function for non-fuel resonance treatment, achieving more than an order-of-magnitude speedup compared to the Tone method; an on-the-fly multigroup equivalence approach to mitigate group condensation errors; and a streaming correction method for pebble-bed homogenization. A proof-of-concept demonstration of on-the-fly group condensation with consistent P0 transport correction was also achieved. The method reproduced direct fine-group solutions with excellent accuracy (eigenvalue errors within 10 pcm and pin-power differences within 0.5%), but due to performance limitations of the current fixed-source solver, improvements to solver efficiency will be addressed in future work. Verification tests were performed on graphite-moderated TRISO-fueled two-dimensional core benchmark problems representing gas-cooled microreactors, heat pipe-cooled microreactors, gas-cooled pebble-bed reactors, and fluoride salt-cooled high-temperature reactors. Across all cases, Griffin showed excellent agreement with Serpent2 continuous energy Monte Carlo solutions: eigenvalue errors within 200 pcm, pin-power root-mean-square errors within 2%, and control rod and drum worth errors less than 2%. It should be noted that, for the benchmark problem, cross section generation contributed less than 3% of the total simulation times. These results demonstrate that Griffin’s online cross section generation capability delivers accurate and efficient reactor physics solutions across a wide spectrum of TRISO-fueled advanced reactor designs. With further improvements to the fine-group fixed-source solver and planned extensions to depletion, transients, and coupled neutron–gamma transport, Griffin will be well-positioned to become a powerful and comprehensive tool for advanced reactor analysis.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2022

Benchmarking Solid-State Batteries Containing Sulfide Separators: Effects of Electrode Composition and Stack Pressure

Integrating sulfide separators into solid-state batteries (SSBs) containing high energy cathodes typically requires one or more materials and engineering solutions including: (i) applying interfacial coatings to mitigate electrolyte decomposition, (ii) applying high stack pressures to form robust solid-solid contacts, and (iii) using alloying anodes to avoid Li dendrite formation. Despite the promise of these approaches, a lack of standardized testing protocols makes it difficult to directly compare results among different studies. To address this problem, the present work benchmarks the performance of SSBs containing β-Li 3 PS 4 (LPS) separators and composite cathodes. By systematically varying the anode/cathode composition and stack pressure, this work demonstrates that cathode design is a major bottleneck for solid-state cells cycled at low rates (<100 µA cm -2 ). Operando stack pressure measurements show that, while mechanical confinement generally promotes higher active material utilization and cycling stability, this strategy alone does not address interfacial reactivity between LPS and high voltage cathodes. Here, these results also demonstrate that stress evolution during cycling is dominated by volume changes at the Li metal anode. Finally, we show that FeS 2 cathodes with moderate operating voltages (<3 V vs Li/Li + ) exhibit superior cycling performance compared to high voltage cathodes by facilitating formation of stable cathode/electrolyte interfaces.

25 ENERGY STORAGE↗
DOE OSTI2021

Synthetic, Realistic Transmission and Distribution Co-Simulation for Voltage Control Benchmarking: Preprint

As distributed energy resources play an increasing role in generation and grid services, control and market paradigms will need to be tested with models that incorporate transmission and distribution systems. The test systems will need to be realistic and accessible for researchers to form conclusions about realistic behaviors and maintain comparability and repeatability of results. These test systems also must have high resolution models of sufficient size for realistic distribution and transmission interaction. This paper demonstrates the tightly coupled co-simulation of realistic synthetic distribution and transmission models for the Austin metropolitan area and how this framework can be used to analyze resultant behaviors from distributed resources and their controls. The comparison of a distribution system run with fixed feed-in voltage is compared to the closely tied distribution-transmission co-simulation. It is determined that even without advanced voltage controls, the feed-in voltage response of the transmission system shows a lower level of voltage excursions. Therefore, control methods for voltage stability should be benchmarked on realistic transmission and distribution co-simulations for more accurate assessment of voltage profiles.

27 ARPA - Advanced Research Projects Agency-Energy↗
DOE OSTI2023

A Reproducible Validation of Algorithms for Estimating Array Tilt and Azimuth from Photovoltaic Power Time Series

In this research, we assess the viability of four different, publicly available algorithms for estimating the azimuth and tilt parameters of solar photovoltaic systems using only the associated AC power time series data and site latitude-longitude coordinates. In this work, we curated a benchmarking data set of 44 fixed-tilt systems, comprising 275 measured AC power inverter data streams, with known azimuth and tilt parameters. Additionally, we isolated test cases in the data set with real- world issues, including shading and clipping, to determine how algorithm performance varies based on the presence of these phenomena. Using this data set for benchmarking, we evaluated the estimated vs. actual system characteristics for each algorithm, as well as the associated algorithm execution time using a standardized benchmarking process. The two highest performing algorithms were the Solar Data Tools and the PVWatts 5- based methods, which both achieved a median absolute error of approximately 5 and 1 degrees for azimuth and tilt, respectively. During run time analysis, the SDT method was approximately 5 times faster than the PVWatts 5-based method, with the median execution time for a stream varying between 6 and 8 seconds vs. a median run time of 31 seconds for the PVWatts 5-based method.

azimuth↗
DOE OSTI2024

Linearised Fokker–Planck collision model for gyrokinetic simulations

We introduce a gyrokinetic, linearised Fokker–Planck collision model that satisfies conservation laws and is accurate at arbitrary collisionalities. The differential test-particle component of the operator is exact; the integral field-particle component is approximated using a spherical harmonic and a modified Laguerre polynomial expansion developed by Hirshman and Sigmar (1976 Phys. Fluids 19 1532). The numerical methods of the implementation in the δf-gyrokinetic code stella (Barnes et al 2019 J. Comput. Phys. 391 365–80) are discussed, and conservation properties of the operator are demonstrated. The collision model is then benchmarked against the collision model of the gyrokinetic solver GS2 in the limiting cases of a reduced test-particle collision operator and energy- and momentum-conserving operator. The accuracy of the full collision model is investigated by solving the parallel Spitzer-Härm problem for the transport coefficients. It is shown that retaining collisional energy flux and higher-order terms in the field-particle operator reduces errors in the transport coefficients from 10%–25% for a simple momentum- and energy-conserving model to under 1%.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2023

A Reproducible Validation of Algorithms for Estimating Array Tilt and Azimuth from Photovoltaic Power Time Series

In this research, we assess the viability of four different, publicly available algorithms for estimating the azimuth and tilt parameters of solar photovoltaic systems using only the associated AC power time series data and site latitude-longitude coordinates. In this work, we curated a benchmarking data set of 44 fixed-tilt systems, comprising 275 measured AC power inverter data streams, with known azimuth and tilt parameters. Additionally, we isolated test cases in the data set with real-world issues, including shading and clipping, to determine how algorithm performance varies based on the presence of these phenomena. Using this data set for benchmarking, we evaluated the estimated vs. actual system characteristics for each algorithm, as well as the associated algorithm execution time using a standardized benchmarking process. The two highest performing algorithms were the Solar Data Tools and the PVWatts 5-based methods, which both achieved a median absolute error of approximately 5 and 1 degrees for azimuth and tilt, respectively. During run time analysis, the SDT method was approximately 5 times faster than the PVWatts 5-based method, with the median execution time for a stream varying between 6 and 8 seconds vs. a median run time of 31 seconds for the PVWatts 5-based method.

algorithm validation↗
DOE OSTI2024

Benchmarking machine learning interatomic potentials via phonon anharmonicity

Abstract Machine learning approaches have recently emerged as powerful tools to probe structure-property relationships in crystals and molecules. Specifically, machine learning interatomic potentials (MLIPs) can accurately reproduce first-principles data at a cost similar to that of conventional interatomic potential approaches. While MLIPs have been extensively tested across various classes of materials and molecules, a clear characterization of the anharmonic terms encoded in the MLIPs is lacking. Here, we benchmark popular MLIPs using the anharmonic vibrational Hamiltonian of ThO 2 in the fluorite crystal structure, which was constructed from density functional theory (DFT) using our highly accurate and efficient irreducible derivative methods. The anharmonic Hamiltonian was used to generate molecular dynamics (MD) trajectories, which were used to train three classes of MLIPs: Gaussian approximation potentials, artificial neural networks (ANN), and graph neural networks (GNN). The results were assessed by directly comparing phonons and their interactions, as well as phonon linewidths, phonon lineshifts, and thermal conductivity. The models were also trained on a DFT MD dataset, demonstrating good agreement up to fifth-order for the ANN and GNN. Our analysis demonstrates that MLIPs have great potential for accurately characterizing anharmonicity in materials systems at a fraction of the cost of conventional first principles-based approaches.

interatomic potentials↗