Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmark testing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 415 records · Page 23

Corrosion Control in Carbon Fiber Reinforced Plastic (CFRP) Composite-Aluminum Closure Panel Hem Joints

This project targeted the implementation of CFRP/Al closure panels as a drop-in replacement for all-aluminum closures used in automobile production today. The project focused on the development of corrosion fundamentals involving CFRP materials and the use of those materials in mixed-material joints. Numerous studies were completed to benchmark current materials and test mixed metal joints. Good correlation was found between outdoor exposures and laboratory cyclic corrosion testing. The insights from the corrosion testing fundamentals and benchmarking phases informed the development of low-cure adhesive and electrocoats and conductive primers for the CFRP. Adhesive and electrocoat materials were successfully developed meeting low-cure targets for bake (150C/10min). These materials were scaled-up and used to produce 5 prototype liftgates to further evaluate CFRP/Al joints in a full-scale part. The lower cure materials enabled mitigation of CTE mismatch and reduced the required bake temperature, resulting in reduced part distortion. Corrosion testing of the liftgates revealed unexpected increases in corrosion relative to the standard bake materials. While rigorous evaluation of the parts was not possible due to numerous complexities associated with prototype testing, the results provided significant insights into the material behavior of CFRP/Al joints. Key learnings from the project included a better understanding of the system complexity (interplay of substrate/primer conductivity, production of full-scale prototypes), advances in coating/adhesive design for low-cure applications involving CFRP, and advancement of corrosion test protocols for CFRP.

36 MATERIALS SCIENCE↗

The MPC&A Questionnaire

The questionnaire is the instrument used for recording performance data on the nuclear material protection, control, and accountability (MPC&A) system at a nuclear facility. The performance information provides a basis for evaluating the effectiveness of the MPC&A system. The goal for the questionnaire is to provide an accurate representation of the performance of the MPC&A system as it currently exists in the facility. Performance grades for all basic MPC&A functions should realistically reflect the actual level of performance at the time the survey is conducted. The questionnaire was developed after testing and benchmarking the material control and accountability (MC&A) system effectiveness tool (MSET) in the United States. The benchmarking exercise at the Idaho National Laboratory (INL) proved extremely valuable for improving the content and quality of the early versions of the questionnaire. Members of the INL benchmark team identified many areas of the questionnaire where questions should be clarified and areas where additional questions should be incorporated. The questionnaire addresses all elements of the MC&A system. Specific parts pertain to the foundation for the facility's overall MPC&A system, and other parts pertain to the specific functions of the operational MPC&A system. The questionnaire includes performance metrics for each of the basic functions or tasks performed in the operational MPC&A system. All of those basic functions or tasks are represented as basic events in the MPC&A fault tree. Performance metrics are to be used during completion of the questionnaire to report what is actually being done in relation to what should be done in the performance of MPC&A functions.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Test Cases for Flutter of the Benchmark Models Rectangular Wings on the Pitch and Plunge Apparatus

The supercritical airfoil was chosen as a relatively modem airfoil for comparison. The BOO12 model was tested first. Three different types of flutter instability boundaries were encountered, a classical flutter boundary, a transonic stall flutter boundary at angle of attack, and a plunge instability near M = 0.9 and for zero angle of attack. This test was made in air and was Transonic Dynamics Tunnel (TDT) Test 468. The BSCW model (for Benchmark SuperCritical Wing) was tested next as TDT Test 470. It was tested using both with air and a heavy gas, R-12, as a test medium. The effect of a transition strip on flutter was evaluated in air. The B64AOlO model was subsequently tested as TDT Test 493. Some further analysis of the experimental data for the BOO12 wing is presented. Transonic calculations using the parameters for the BOO12 wing in a two-dimensional typical section flutter analysis are given. These data are supplemented with data from the Benchmark Active Controls Technology model (BACT) given and in the next chapter of this document. The BACT model was of the same planform and airfoil as the BOO12 model, but with spoilers and a trailing edge control. It was tested in the heavy gas R-12, and was instrumented mostly at the 60 per cent span. The flutter data obtained on PAPA and the static aerodynamic test cases from BACT serve as additional data for the BOO12 model. All three types of flutter are included in the BACT Test Cases. In this report several test cases are selected to illustrate trends for a variety of different conditions with emphasis on transonic flutter. Cases are selected for classical and stall flutter for the BSCW model, for classical and plunge for the B64AOlO model, and for classical flutter for the BOO12 model. Test Cases are also presented for BSCW for static angles of attack. Only the mean pressures and the real and imaginary parts of the first harmonic of the pressures are included in the data for the test cases, but digitized time histories have been archived. The data for the test cases are available as separate electronic files. An overview of the model and tests is given, the standard formulary for these data is listed, and some sample results are presented.

Robert M Bennett↗

Implementation of a First-Order Quadratic Program Solver in C

This paper details a translation of a first order quadratic program (QP) solver from MATLAB to C. NASA could use this QP solver to generate online flight path trajectories for powered descent vehicles during landing. Over 12 weeks, the team designed, implemented, and tested two iterations of the QP solver for accuracy and runtime on 104 benchmark QP tests. The final iteration was 541.07% faster than the first, handling most tests in under one second. Additionally, it solved four more QP tests for N≥1383, and all outputs for cost and D_x matched the MATLAB reference values.

Optimization↗

CFD validation in OECD/NEA t-junction benchmark.

When streams of rapidly moving flow merge in a T-junction, the potential arises for large oscillations at the scale of the diameter, D, with a period scaling as O(D/U), where U is the characteristic flow velocity. If the streams are of different temperatures, the oscillations result in experimental fluctuations (thermal striping) at the pipe wall in the outlet branch that can accelerate thermal-mechanical fatigue and ultimately cause pipe failure. The importance of this phenomenon has prompted the nuclear energy modeling and simulation community to establish a benchmark to test the ability of computational fluid dynamics (CFD) codes to predict thermal striping. The benchmark is based on thermal and velocity data measured in an experiment designed specifically for this purpose. Thermal striping is intrinsically unsteady and hence not accessible to steady state simulation approaches such as steady state Reynolds-averaged Navier-Stokes (RANS) models.1 Consequently, one must consider either unsteady RANS or large eddy simulation (LES). This report compares the results for three LES codes: Nek5000, developed at Argonne National Laboratory (USA), and Cabaret and Conv3D, developed at the Moscow Institute of Nuclear Energy Safety at (IBRAE) in Russia. Nek5000 is based on the spectral element method (SEM), which is a high-order weighted residual technique that combines the geometric flexibility of the finite element method (FEM) with the tensor-product efficiencies of spectral methods. Cabaret is a 'compact accurately boundary-adjusting high-resolution technique' for fluid dynamics simulation. The method is second-order accurate on nonuniform grids in space and time, and has a small dispersion error and computational stencil defined within one space-time cell. The scheme is equipped with a conservative nonlinear correction procedure based on the maximum principle. CONV3D is based on the immersed boundary method and is validated on a wide set of the experimental and benchmark data. The numerical scheme has a very small scheme diffusion and is the second and the first order accurate in space and time, correspondingly. We compare and contrast simulation results for three computational fluid dynamics codes CABARET, Conv3D, and Nek5000 for the T-junction thermal striping problem that was the focus of a recent OECD/NEA blind benchmark. The corresponding codes utilize finite-difference implicit large eddy simulation (ILES), finite-volume LES on fully staggered grids, and an LES spectral element method (SEM), respectively. The simulations results are in a good agreement with experimenatl data. We present results from a study of sensitivity to computational mesh and time integration interval, and discuss the next steps in the simulation of this problem.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

IER 480: TEX-Pu Benchmark (PU-MET-THERM-004) to Test Polyethylene and Lucite Thermal Scattering Laws [Slides]

This presentation finds that PMT-004 has four new benchmark cases highly sensitive to PE TSL (2 cases) and PMMA TSL (2 Cases). PE cases were well predicted using MCNP6.2 and ENDF/B-VIII.0. PMMA cases overpredicted by approximately 0.6-0.7% in k eff at 20°C. Accepted into 2023 version of the ICSBEP Handbook. Temperature had a large impact on reactivity of the critical configuration. Implications for validation work for thermal cases- need to adjust TSL data to correct temperature as it can have hundreds of pcm effects for a few °C. Future thermal experiments should try and measure reactivity at multiple temperatures to aid in data testing and benchmark adjustment.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

SU-G-BRB-02: An Open-Source Software Analysis Library for Linear Accelerator Quality Assurance

Purpose: Routine linac quality assurance (QA) tests have become complex enough to require automation of most test analyses. A new data analysis software library was built that allows physicists to automate routine linear accelerator quality assurance tests. The package is open source, code tested, and benchmarked. Methods: Images and data were generated on a TrueBeam linac for the following routine QA tests: VMAT, starshot, CBCT, machine logs, Winston Lutz, and picket fence. The analysis library was built using the general programming language Python. Each test was analyzed with the library algorithms and compared to manual measurements taken at the time of acquisition. Results: VMAT QA results agreed within 0.1% between the library and manual measurements. Machine logs (dynalogs & trajectory logs) were successfully parsed; mechanical axis positions were verified for accuracy and MLC fluence agreed well with EPID measurements. CBCT QA measurements were within 10 HU and 0.2mm where applicable. Winston Lutz isocenter size measurements were within 0.2mm of TrueBeam’s Machine Performance Check. Starshot analysis was within 0.2mm of the Winston Lutz results for the same conditions. Picket fence images with and without a known error showed that the library was capable of detecting MLC offsets within 0.02mm. Conclusion: A new routine QA software library has been benchmarked and is available for use by the community. The library is open-source and extensible for use in larger systems.

43 PARTICLE ACCELERATORS↗

AI Model Benchmarking for Nonproliferation Applications: Steel Thread Benchmarking Task Force Technical Report (Rev. 2)

Steel Thread is a NA-22 venture that seeks to build trustworthy, reliable AI models that can be used in a wide variety of nonproliferation tasks. A key aspect of building these models is developing appropriate benchmarks and evaluation methods, which will enable the venture to identify and adapt models to provide the most value in the nonproliferation domain. Benchmarks must be relevant to key tasks in this domain, such as question answering, information retrieval, document summarization and classification, consensus analysis, and image and data analysis. This report 1) provides an overview of benchmark design, evaluation, and challenges; 2) reviews a variety of open benchmarks, with a focus on language models and tasks; and 3) identifies benchmarks that are most relevant to Steel Thread. This report is intended to serve as a basis for further efforts to classify and evaluate benchmarks and their correlation with success on nonproliferation-specific tasks. The Steel Thread venture has defined benchmarks to be a particular combination of a dataset (or datasets) and a metric (or metrics) conceptualized as representing one or more specific tasks or sets of abilities for a specific modality. It is adopted by a research community as a shared framework for comparing methods.1 It includes 1) Data: Labeled (a designated subset not used for training, which could be all the data), 2) Metric: A way to quantify performance, 3) Task/Ability: What the benchmark is testing, 4) Protocol: A structured and repeatable evaluation process, 5) Baseline/Reference Model: For comparison; could be statistical, rule-based, SME-derived, or another model, and 6) Maintenance Plan: to update with new information over time; important for long-term utility. For further clarity, the definition includes what a benchmark, in this context, is not. It is not a corpus of training data, specific to a model (it is intended to apply to a range of models), a universal evaluation of performance, a guarantee that the ‘top’ model on the leaderboard will be the best fit for every specific use case, an all-encompassing proof of a model’s universal quality, nor is it a one-size-fits-all measure of success. It does not cover every real-world constraint (like operational, ethical, or cost considerations), a systems integration test, or a unit test. This definition was inspired by and resulted from discussions within the Steel Thread Benchmarking Task Force. This group was formed to define what we would mean as a benchmark within Steel Thread but persisted as the need to develop a thorough understanding of the large and expanding existing benchmarking space. This technical report is a result of the group’s divide and conquer approach to exploring this space. The release of benchmarks might not be progressing as quickly as model development, but it is moving very fast, as many benchmarks quickly become saturated, when state-of-the-art models score so close to the benchmark’s ceiling that their results are virtually indistinguishable. At that point, the test no longer differentiates between new systems, so researchers usually stop reporting scores as the benchmark no longer informs about improvements from the next generation of models. In the OpenAI announcement of GPT-5, they reported results on six flagship public benchmarks (AIME 2025, SWE-bench Verified, Aider Polyglot, MMMU, HealthBench Hard, GPQA) but the full system-card covers roughly thirty-five separate evaluations, comprising hundreds of test task items in total. There have been some efforts to summarize benchmarks in specific fields, like for text-to-image generation, but these surveys have had a narrow methodology scope. Therefore, a comprehensive survey of all benchmarks or even all benchmarks that could be relevant to Steel Thread is outside of the scope of this report. We chose some specific benchmarks to investigate in detail.

97 MATHEMATICS AND COMPUTING↗

Toward a Benchmark for Multi-Threaded Testing Tools

Looking for intermittent bugs is a problem that has been getting prominence in testing. Multi-threaded code is becoming very common, mostly on the server side. As there is no silver bullet solution, research focuses on a variety of partial solutions. We outline a road map for combining the research on the different disciplines of testing multi-threaded programs and on evaluating its quality. The project goals are to create a benchmark that can be used to evaluate different solutions, to create a framework with open API's that enables combining techniques in the multithreading domain, and to create a focus for the research in this area around which a community of people who try to solve similar problems with different techniques, could congregate. The benchmark, apart from containing programs with documented bugs, includes other artifacts, such as traces, that are used for evaluating some of the technologies. We have started creating such a bench mrk and detail the lesson learned in the process. The framework will enable technology developers, for example, race detectors, to concentrate on their components and use other ready made components, (e.g., instrumentor) to create a testing solution.

Eytani, Yaniv↗

AREVA NP's industrial CFD single-phase methodology and applications for nuclear fuel

During the course of the last decade, AREVA NP has developed Computational Fluid Dynamics (CFD) methods capable of accurately predicting flow fields and thermal mixing within fuel bundles, as well as pressure losses and hydrodynamic forces on fuel assembly components such as spacer grids, lower and upper nozzles. The present work summarizes the evolution of AREVA NP's single-phase CFD methodology and the extensive validation efforts, with focus on improving confidence level in CFD results. Currently, it is possible to produce models for large sections of a fuel assembly and produce CFD simulation results with an uncertainty proven to be similar to the corresponding measurement uncertainty. Extensive validation has been conducted against in-house data obtained for a wide range of geometries representative of actual hardware, in multiple test facilities, with various measurement techniques. In addition to in-house validation data, AREVA NP has also exercised its best-practices against international benchmarks; blind test results were in very good agreement with the data. The current high level of confidence in single-phase CFD predictions allows investigating a wide variety of physics in Pressurized Water Reactor (PWR) as well as Boiling Water Reactor (BWR) fuel assemblies. CFD is used as a design and optimization tool as well as a supplement to regular testing to improve the quality of tests and optimize hardware design features for specific targets. CFD played a major role in the design of AREVA NP's latest fuel assembly designs (ATRIUM{sup TM1} 11 and GAIA). In this paper we first illustrate AREVA NP's comprehensive validation database for single-phase CFD (pressure drop, flow field), then provide several examples of applications: virtual testing for parts optimization and support of core operation. (authors)

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Open Source Evaluation Framework for Solar Forecasting

The Solar Forecast Arbiter is an open-source evaluation framework for solar forecasting. The framework enables evaluations of solar irradiance, solar power, and net-load forecasts that are impartial, repeatable and auditable. The Solar Forecast Arbiter addresses stakeholder-informed use cases including evaluation of forecast skill, comparisons to reference data sets, private forecast trials, and evaluation of probabilistic forecast skill. The framework includes a data validation toolkit, reference data sources, data privacy protocols, and benchmark forecast capabilities for intra-hour and day ahead forecast horizons. Reports and metrics communicate the relative merits of the test and benchmark forecasts. The reports are created from standardized templates and include graphics for qualitatively evaluating deterministic and probabilistic forecasts and standard metrics for quantitatively evaluating forecasts. The Solar Forecast Arbiter is designed to support all solar forecasting stakeholders, including Solar Forecasting 2 Topic Area 2 and Topic Area 3 teams.

14 SOLAR ENERGY↗

Laboratory measurements of large-scale carbon sequestration flows in saline reservoirs

Brine saturated with CO{sub 2} is slightly denser than the original brine causing it to sink to the bottom of a saline reservoir where the CO{sub 2} is safely sequestered. However, the buoyancy of pure CO{sub 2} relative to brine drives it to the top of the reservoir where it collects underneath the cap rock as a separate phase of supercritical fluid. Without additional processes to mix the brine and CO{sub 2}, diffusion in this geometry is slow and would require an unacceptably long time to consume the pure CO{sub 2}. However, gravity and diffusion-driven convective instabilities have been hypothesized that generate enhanced CO{sub 2}-brine mixing promoting dissolution of CO{sub 2} into the brine on time scale of a hundred years. These flows involve a class of hydrodynamic problems that are notoriously difficult to simulate; the simultaneous flow of mUltiple fluids (CO{sub 2} and brine) in porous media (rock or sediment). The hope for direct experimental confirmation of simulations is dim due to the difficulty of obtaining high resolution data from the subsurface and the high pressures ({approx}100 bar), long length scales ({approx}100 meters), and long time scales ({approx}100 years) that are characteristic of these flows. We have performed imaging and mass transfer measurements in similitude-scaled laboratory experiments that provide benchmarks to test reservoir simulation codes and enhance their predictive power.

58 GEOSCIENCES↗

Performance modeling & simulation of complex systems (A systems engineering design & analysis approach)

Modeling of the Multi-mission Image Processing System (MIPS) will be described as an example of the use of a modeling tool to design a distributed system that supports multiple application scenarios. This paper examines: (a) modeling tool selection, capabilities, and operation (namely NETWORK 2.5 by CACl), (b) pointers for building or constructing a model and how the MIPS model was developed, (c) the importance of benchmarking or testing the performance of equipment/subsystems being considered for incorporation the design/architecture, (d) the essential step of model validation and/or calibration using the benchmark results, (e) sample simulation results from the MIPS model, and (f) how modeling and simulation analysis affected the MIPS design process by having a supportive and informative impact.

Hall, Laverne↗

DeepBench: A simulation package for physical benchmarking data

We introduce **DeepBench**, a python library that generates simple simulated image data from first principles, such as basic geometric shapes and astronomical objects. These data are highly valuable for developing (calibration, testing, and benchmarking) statistical and machine learning models because they make it possible to connect the final data product to physically interpretable inputs. This software includes tools to curate and store the datasets to maximize reproducibility.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Well-balanced Component-wise Scheme for Shallow Water System

This paper presents a well-balanced numerical scheme for solving free surface flows involving wetting and drying. The proposed algorithm combines a component-wise approach with hydrostatic reconstruction strategy to compute flows over wet or dry surfaces and to satisfy the steady state condition of still water. The robustness of the proposed scheme is verified under several benchmark hydraulic tests.

42 ENGINEERING↗

LCIO

Performance of file systems shift during their life cycles. Evaluating this performance change over time is not trivial. Complexity arises in the interplay between external (i.e. application I/O workloads) and internal (i.e. the filesystem state) factors. Many benchmarks can test how a filesystem performs at the current snapshot state, but to observe the change over time necessitates that the filesystem state mutate (age) between benchmark runs. For a large-scale HPC parallel filesystem, the sheer scale and amount of interacting components during I/O operations magnify these challenges. LCIO addresses the question - how will the filesystem perform at different stages of its life cycle? LCIO is a synthetic benchmark, which provides the file system aging process to increase the amount of information that existing benchmarks like IOR and MDTest yield, as well as provide additional points of data that will be useful to system architects and engineers.

Bachstein, Matthew↗

Thermal scattering law of C 2 H 4 n : Integrating experimental data with DFT calculations

Improvements in determination of the thermal scattering law of moderator materials (measuring, calculating and validating) are important for accurate prediction of neutron thermalization in nuclear systems. In this paper a methodology for producing thermal scattering libraries from the experimental data for polyethylene C 2 H 4 n is discussed. Double differential scattering cross section (DDSCS) experiments were performed at the Spallation Neutron Source of Oak Ridge National Laboratory (SNS ORNL). New scattering kernel evaluations, based on phonon spectrum for C 2 H 4 n , are created using the NJOY2016 code. Two different methods were used: direct and indirect geometry neutron scattering at ARCS and SEQUOIA, and VISION instruments, respectively, where the phonon spectrum was derived from the dynamical structure factor S(Q, ω ) obtained from the measured DDSCS. In order to compare and validate the newly created library, the experimental setup was simulated using MCNP6.1. Compared with the current ENDF/B-VII.1, the resulting RPI C 2 H 4 n libraries improved both double differential scattering and total scattering cross sections. A set of criticality benchmarks containing C 2 H 4 n from HEU-MET-THERM resulted in an overall improved calculation of K eff , although the libraries should be tested against benchmarks more sensitive to C 2 H 4 n . The DFT + oClimax method is used and is shown to be most comprehensive method for analysis of moderator materials. Finally, the importance of DFT + oClimax method lies in the fact that it can be validated against all data measured at VISION, ARCS and SEQUOIA, and experimental total scattering cross section measurements.

74 ATOMIC AND MOLECULAR PHYSICS↗