SEARCH · Search NASA

Results for “Benchmarking”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

All sources NASA NTRS DOE OSTI

DOE OSTI2025

Benchmarking quantum computers

The rapid pace of development in quantum computing technology has sparked a proliferation of benchmarks to assess the performance of quantum computing hardware and software. However, not all benchmarks are of equal merit. Good ones empower scientists, engineers, programmers and users to understand the power of a computing system, whereas bad ones can misdirect research and inhibit progress. In this Perspective, we survey the science of quantum computer benchmarking. Here, we discuss the role of benchmarks and benchmarking and how good benchmarks can drive and measure progress towards the long-term goal of useful quantum computations, known as quantum utility. We explain how different kinds of benchmark quantify the performance of different parts of a quantum computer, discuss existing benchmarks, examine recent trends in benchmarking, and highlight important open research questions in this field.

Proctor, Timothy James [Sandia National Laboratori↗

DOE OSTI2023

Benchmarking for AI for Science

AI has been instrumental for recent developments in a number of domains of the sciences. With several hundred machine learning (ML) algorithms and models, and numerous AI-specific hardware platforms, a common quest for all scientists working on AI for Science is around the selection of machine learning algorithm(s) to solve their domain-specific scientific problems. A number of different initiatives around AI Benchmarking have been set up and have been useful in understanding the benefits of different ML algorithms for different tasks.However, with the majority of these AI Benchmarking initiatives focusing on the conventional notions of benchmarking, where the focus is purely runtime performance (such as training time or inference time), their suitability for benchmarking different ML algorithms for solving scientific problems has been viewed as a performance problem even though both are hardly the same. To make reasonable, explainable, and justifiable advancements in science using AI, it is critical to focus on the merits of these algorithms in handling different domain science problems. In other words, more emphasis must be given on Benchmarking for AI for Science than AI Benchmarking. The vision of the former is not only to assess the performance of ML algorithms, but also to assess, and understand the benefits and merits of different ML algorithms in handling scientific problems. Benchmarking for AI for Science, instead of pure performance focused AI Benchmarking, has several benefits: (i) it has the potential to offer advances in the sciences, much more rapidly than through pure performance-based AI methods, (ii) it will encourage the community to focus on developing better domain-specific AI techniques, particularly given the provision for being able to benchmark different techniques, and (iii) it will encourage hardware manufacturers to focus on developing science-specific hardware subsystems.

Thiyagalingam, Jeyan↗

DOE OSTI2018

Entity Resolution at Large Scale: Benchmarking and Algorithmics

We seek scalable benchmarks for entity resolution problems. Solutions to these problems range from trivial approaches such as string sorting to sophisticated methods such as statistical relational learning. The theoretical and practical complexity of these approaches varies widely, so one of the primary purposes of a benchmark will be to quantify the trade-off between solution quality and runtime. We are motivated by the ubiquitous nature of entity resolution as a fundamental problem faced by any organization that ingests large amounts of noisy text data. A benchmark is typically a rigid specification that provides an objective measure usable for ranking implementations of an algorithm. For example the Top500 and HPCG500 bench- marks rank supercomputers based on their performance of dense and sparse linear algebra problems (respectively). These two benchmarks require participants to report FLOPS counts attainable on various machines. Our purpose is slightly different. Whereas the supercomputing benchmarks mentioned above hold algorithms constant and aim to rank machines, we are primarily interested in ranking algorithms. As mentioned above, entity resolution problems can be approached in completely different ways. We believe that users of our benchmarks must decide what sort of procedure to run before comparing implementations and architectures. Eventually, we also wish to provide a mechanism for ranking machines while holding algorithmic approach constant . Our primary contributions are parallel algorithms for computing solution quality mea- sures per entity. We find in some real datasets that many entities are quite easy to resolve while others are difficult, with a heavy skew toward the former case. Therefore, measures such as global confusion matrices, F measures, etc. do not meet our benchmarking needs. We design methods for computing solution quality at the granularity of a single entity in order to know when proposed solutions do well in difficult situations (perhaps justifying extra computational), or struggling in easy situations. We report on progress toward a viable benchmark for comparing entity resolution algo- rithms. Our work is incomplete, but we have designed and prototyped several algorithms to help evalute the solution quality of competing approaches to these problems. We envision a benchmark in which the objective measure is a ratio of solution quality to runtime.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Comparison Study of Upper Subcritical Limits Derived Using Sensitivity/Uncertainty Tools Case Studies of Benchmarks and Applications

Neutron transport methods used to establish subcriticality require validation by comparison to critical experiments considered to be benchmarks. Whisper is a sensitivity/uncertainty analysis tool developed to assist with the task of validation in nuclear criticality safety. Details on the Whisper methodology can be found in References 1-3 on the MCNP ® reference collection website at https://mcnp.lanl.gov. Whisper-1.0 was originally developed in 2014 and used to assist with nuclear criticality safety validation at Los Alamos National Laboratory. Whisper was upgraded in 2016 to Whisper-1.1 and prepared for release with MCNP6.2. Whisper contains a library of over 1100 critical experiment benchmarks and quantifies neutronic similarity of an application to benchmarks in the library. Using highest similarity benchmarks, Whisper computes a calculational margin (CM) encompassing of the worst-case bias and bias uncertainty at a 99% confidence level for each application. In addition, portions of the margin of subcriticality (MOS) for nuclear data uncertainty and potential code errors are computed. The baseline upper subcritical limit (USL) computed by Whisper is comprised of the CM, MOS nuclear data , and MOS code errors . The Whisper baseline USL is absent a portion of the MOS due to the area of application, which is applied based upon judgment by the criticality safety analyst. An objective of this paper is to present the baseline USL, CM and portions of the MOS as computed by Whisper for comparison with similar sensitivity/uncertainty tools, such as those used by IRSN and ORNL. The initial comparison involves four critical experiment benchmarks: HEU-MET-FAST-013-001, HEU-SOLTHERM-001-008, PU-MET-FAST-022-001, AND PU-SOL-THERM-001-001, which have been: 1. modeled independently by LANL, IRSN, and ORNL based upon information provided in the ICSBEP Handbook, 2. are common in S/U libraries for LANL, IRSN, and ORNL, 3. span a range of energy spectrum and fissile material, and 4. taken as applications for the purposes of this study and therefore excluded from use as a benchmark for calculating the upper subcritical limit. Results presented in this paper have been computed using covariance data for all isotopes in ENDF/BVII.0 using a 44-group energy structure. Benchmarks in the Whisper library were run in MCNP6.2 using 100,000 neutrons per cycle, skipping 100 cycles for 500 active cycles. Reference 10 also compares the results for baseline USL with an order of magnitude greater neutrons, using the same total number of cycles with 1,000,000 neutrons per cycle. Subsequent to the results presented in Reference 10 changes were made to the benchmark library, as discussed in Reference 11. Newer results using the revised benchmark library are presented herein.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2021

Sensitivity and Uncertainty of the IFR-1 BISON Benchmark

The fuel performance code BISON is being used to evaluate metallic fuel for a new fast-spectrum test reactor called the Versatile Test Reactor, which is being considered by the US Department of Energy. To quantify the accuracy of BISON predictions, researchers at Oak Ridge National Laboratory have been developing a series of benchmarks based on legacy metallic fuel experiments. As part of this effort, the sensitivity of BISON predictions to variations in model inputs and the uncertainties associated with BISON predictions must be established. This report summarizes efforts to perform a comprehensive sensitivity analysis (SA) and uncertainty quantification (UQ) on a benchmark based on the IFR-1 experiment. For the SA, at least one input was chosen from every BISON model and physics module used in the benchmark. The inputs were varied individually in a series of BISON simulations. The resulting variations in benchmark predictions were normalized to calculate sensitivities. The strongest sensitivities were identified and used to inform input selections for the UQ. The UQ was performed using the Monte Carlo UQ method. A literature review was conducted to estimate uncertainty distributions for the selected inputs, and values were sampled randomly from each distribution in a series of BISON simulations. Variations in the benchmark predictions were used to estimate uncertainty distributions and confidence intervals. It was found that nearly 100% of benchmark predictions matched the corresponding legacy values within the confidence intervals. However, this is at least partially because of the wide confidence intervals associated with the benchmark predictions. The uncertainty contributions of assumptions in the benchmark, experimental uncertainties, and BISON models were quantified. Some analysis was performed to identify inputs that contributed to the uncertainties. Finally, recommendations are made for future benchmark development and future BISON development.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2021

LANL Critical Benchmark Comparison Study and Subsequent Revision for Cases Involving LEU and MIX

As part of an international collaboration within the DOE Nuclear Criticality Safety Program (NCSP), LANL is involved in a comparison study to quantify differences in k-effective results from neutron transport simulations of critical benchmark experiments. The DOE NCSP Mission and Vision details the activity in which the French Institut de Radioprotection et de Sûreté Nucléaire (IRSN) leads the study with LANL, ORNL, and LLNL to compare results of various neutron transport codes and nuclear data libraries for ICSBEP benchmarks held in common by the entities. The task statement from the DOE NCSP Five-Year Execution Plan is given as: "The proposal is for IRSN to lead a new intercomparison based on the MORET code with the latest JEFF-3.3 data and ENDF/B-VIII.0 data, when available, using their existing comprehensive selection of 2,714 benchmarks and collate their results together with those from LLNL (COG), LANL (MCNP) and ORNL (SCALE). Due to the large number of benchmarks involved, this effort is envisioned to take three years with an additional year for IRSN to complete a summary report. The benchmark development will be performed independently to minimize modeling errors through discovery and resolution of discrepant results. A summary report will be generated (led by IRSN) to document the results of this study." This report documents results obtained for revisions made to cases involving Intermediate Enriched Uranium (IEU) and a mixture of Pu and Uranium (MIX), with a focus on the changes made to LANL benchmarks modeled with MCNP6 using ENDF/B-VII.1 nuclear data that appeared to have discrepant results when compared with results of other codes. Feedback was used to pinpoint review of particular benchmark input files and to revise them when necessary. This report documents the results of review and revision of specific benchmarks highlighted as possibly discrepant in the comparison study. In addition, there is an effort tied to this work involving collaboration between LANL XCP and NCS Divisions in the development of a shared review/revision procedure and use of a new benchmark repository.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2019

Criticality Accident Alarm System Shielding Benchmark: Integral Experiment Request 498, Critical Engineering Decision 1 Report

This report supports Critical/Subcritical Experiment Decision 1 (CED-1) for Integral Experiment Request (IER) 498, a shielding benchmark produced using the Godiva IV critical assembly at the National Critical Experiments Research Center (NCERC). Shielding benchmarks are datasets collected to provide a comparison for computer models that simulate the effect of shielding on radiation transmission from a source to a detector. The purpose of a CED-1 report is to produce a conceptual design for an integral experiment and to identify the long lead-time items. The design presented in this report favors certain detailed design options such as material selections. However, alternative options can be selected as the project evolves. Dimensions are specified only as needed to evaluate feasibility and performance. Development of Criticality Accident Alarm System (CAAS) shielding benchmarks supports two main programmatic needs. The first need is to improve confidence in models of facilities that house nuclear material handling operations. Better models can support effective CAAS detector placement when criticality may result in dose, and it enables more accurate analysis of criticality accident impacts at various locations in a facility. A key quantity that can be verified is the secondary gamma production caused by neutron interactions with various materials. The second programmatic need supported by shielding benchmarks is the ongoing development of methods being integrated into nuclear simulation codes such as the Monte Carlo N-Particle (MCNP) transport code and the SCALE code system used in efforts funded by the Nuclear Criticality Safety Program (NCSP). Previous work performed under the International Criticality Safety Benchmark Experimental Program (ICSBEP) to produce shielding benchmark data is primarily limited to a set of measurements at the SILENE critical assembly at the Valduc facility in France. The experiment used an open configuration and a burst mode operation, which limited detector options to thermoluminescent dosimeters (TLDs) and activation foils. Because TLDs do not record gamma spectra, it was difficult to verify the gamma energy spectrum from the experiment. In this work, the Godiva IV assembly will be used because it is a fast system capable of producing a wide range of source strengths, including super-prompt critical bursts. The room return for the facility is expected to be substantial and challenging to characterize, so a more closed, shielded experimental configuration is considered. Furthermore, because the physics of radiation transport outside the assembly does not change between the burst mode and the delayed critical mode, it is anticipated that most measurements will be made with the Godiva IV assembly in delayed critical mode, operating within a range that will allow electronic counters to collect accurate data. Scoping calculations concluded that a workable room return shield is comprised of 40 cm of 5% borated polyethylene backed by 7 cm of lead. This configuration was shown to produce a ratio of signal–to–room return of over 100 for up to 2 mean free paths of shield sample material, and over 10 for up to 4 mean free paths of shield sample material, ensuring that useful results can be obtained. With no shielding sample in the room return shield, it was shown that the neutron and gamma spectra do not appreciably change with a change in room size. This scoping calculation does not preclude the use of tungsten powder in a binder, some other effective gamma shielding material, or a hybrid design. Dose rates for various source intensities at an unshielded detector are listed herein to inform detector selection during CED-2, and calibration chains are identified. Preliminary estimates of costs and schedules are outlined but remain to be verified. Values could double or triple as estimates become more accurate and are principally included here to demonstrate feasibility. Preliminary estimates indicate that the apparatus can be installed within a feasible schedule and budget. Two measurements per day are assumed. Producing a shielding benchmark at NCERC using the Godiva IV assembly has been shown to be feasible, with manageable uncertainties and risks. A clear, continuing need for shielding benchmark data remains present, and this report identifies and evaluates a method for creating integrated shielding benchmark datasets.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2025

FAIR Surrogate Benchmarks Supporting AI and Simulation Research (Final Report)

Computational Science is being revolutionized by integrating AI and simulation and, in particular, by deep learning surrogate models that can replace all or part of traditional large‐scale HPC computations. Such surrogates can achieve remarkable performance improvements, as much as several orders of magnitude, and save both compute time and energy. The Surrogate Benchmark Initiative (SBI) project creates a community repository and FAIR (Findable, Accessible, Interoperable, and Reusable) data ecosystem for HPC application surrogate benchmarks. The SBI team comes from Argonne National Laboratory (ANL), Indiana University (IU), Rutgers University, the University of Tennessee, Knoxville (UTK), and the University of Virginia (UVA). SBI repositories include data, code, and all relevant collateral artifacts that the science and engineering community need to use and reuse these data sets and surrogates. SBI repositories generate active research from both the participants in SBI and the broad community of AI and domain scientists. This project develops surrogates that use several different neural nets to learn and quickly infer the results of simulations and data systems and captures them as surrogate benchmarks with a rich set of metadata covering: Data; Model; Metrics specification; Machine specification; and Science, Speed, and Power Results. We research FAIR metadata for these benchmarks. We develop application surrogate examples as benchmarks across many fields (ANL, UTK, IU, UVA). We also study non-Surrogate benchmarks that have many common features and similar issues as regards FAIRness. We work with MLCommons (UVA, UTK), which is a major machine learning benchmarking activity where we get metadata ontologies, software, and benchmarks, Benchmarks have datasets, models, and metadata and they need a technical framework developed by UTK and Rutgers and deployed by UVA. We study features of Surrogates including performance, training set size, and uncertainty quantification (Rutgers, UVA and IU).

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

A volumetric framework for quantum computer benchmarks

We propose a very large family of benchmarks for probing the performance of quantum computers. We call them volumetric benchmarks (VBs) because they generalize IBM's benchmark for measuring quantum volume \cite{Cross18}. The quantum volume benchmark defines a family of square circuits whose depth d and width w are the same. A volumetric benchmark defines a family of rectangular quantum circuits, for which d and w are uncoupled to allow the study of time/space performance trade-offs. Each VB defines a mapping from circuit shapes — ( w , d ) pairs — to test suites C ( w , d ) . A test suite is an ensemble of test circuits that share a common structure. The test suite C for a given circuit shape may be a single circuit C , a specific list of circuits { C 1 … C N } that must all be run, or a large set of possible circuits equipped with a distribution P r ( C ) . The circuits in a given VB share a structure, which is limited only by designers' creativity. We list some known benchmarks, and other circuit families, that fit into the VB framework: several families of random circuits, periodic circuits, and algorithm-inspired circuits. The last ingredient defining a benchmark is a success criterion that defines when a processor is judged to have ``passed'' a given test circuit. We discuss several options. Benchmark data can be analyzed in many ways to extract many properties, but we propose a simple, universal graphical summary of results that illustrates the Pareto frontier of the d vs w trade-off for the processor being benchmarked.

97 MATHEMATICS AND COMPUTING↗

NASA NTRSDate not supplied

Creating Benchmark Data for Artificial Intelligence and Machine Learning Space Biology Research

To identify an appropriate AI/ML approach for a specific problem, the best practice is to measure algorithm performance through the benchmarking process. A scientific benchmark consists of an AI-ready dataset and a reference implementation on a specific scientific question. The NASA Science Mission Directorate (SMD) has started the “Benchmark Initiative for AI/ML to create scientific benchmark datasets in three applications: 1) scientific benchmarking, which finds the best algorithm for a specific problem; 2) application benchmarking, which measures algorithm performance against a set of parameters; and 3) system benchmarking, which evaluates performance of hardware and software architecture. Currently, there are no standardized datasets available to benchmark AI/ML algorithms in the domain of space biology. In this work, we constructed two AI/ML-ready biological datasets from experiments in space-flown mice: cellular imaging and RNA-seq. First, radiation-exposed immune cells harbor DNA damage foci that can be fluorescently marked to visualize the amount of damage following exposure to ionizing radiation. However, such large datasets are difficult to analyze visually, due to imaging inconsistencies and human bias, and classical image processing approaches can fail on imaging artifacts. AI/ML are therefore exciting alternative, providing the speed of machines and the accuracy of humans. We have made this dataset available at https://registry.opendata.aws/bps_microscopy/. Second, high-throughput nucleic acid sequencing (DNA-seq, RNA-seq) has become widespread in biomedical research due to the growing availability and affordability of these assays. However, most sequencing datasets suffer from high dimensionality and low sample count. In this work, we used a generative adversarial network to synthesize a standardized, AI-ready, publicly available benchmark dataset for space biology RNA-seq data with sufficient space-flown and ground control mouse liver samples from NASA GeneLab. This dataset is available at https://registry.opendata.aws/bps_rnaseq/. These datasets are now fully open the Space Biology community to test their favorite AI/ML approaches.

James Casaletto↗

DOE OSTI2016

Benchmark problems for numerical implementations of phase field models

Here, we present the first set of benchmark problems for phase field models that are being developed by the Center for Hierarchical Materials Design (CHiMaD) and the National Institute of Standards and Technology (NIST). While many scientific research areas use a limited set of well-established software, the growing phase field community continues to develop a wide variety of codes and lacks benchmark problems to consistently evaluate the numerical performance of new implementations. Phase field modeling has become significantly more popular as computational power has increased and is now becoming mainstream, driving the need for benchmark problems to validate and verify new implementations. We follow the example set by the micromagnetics community to develop an evolving set of benchmark problems that test the usability, computational resources, numerical capabilities and physical scope of phase field simulation codes. In this paper, we propose two benchmark problems that cover the physics of solute diffusion and growth and coarsening of a second phase via a simple spinodal decomposition model and a more complex Ostwald ripening model. We demonstrate the utility of benchmark problems by comparing the results of simulations performed with two different adaptive time stepping techniques, and we discuss the needs of future benchmark problems. The development of benchmark problems will enable the results of quantitative phase field models to be confidently incorporated into integrated computational materials science and engineering (ICME), an important goal of the Materials Genome Initiative.

36 MATERIALS SCIENCE↗

DOE OSTI2016

Estimating the Need for Palliative Radiation Therapy: A Benchmarking Approach

Purpose: Palliative radiation therapy (PRT) benefits many patients with incurable cancer, but the overall need for PRT is unknown. Our primary objective was to estimate the appropriate rate of use of PRT in Ontario. Methods and Materials: The Ontario Cancer Registry identified patients who died of cancer in Ontario between 2006 and 2010. Comprehensive RT records were linked to the registry. Multivariate analysis identified social and health system-related factors affecting the use of PRT, enabling us to define a benchmark population of patients with unimpeded access to PRT. The proportion of cases treated at any time (PRT{sub lifetime}), the proportion of cases treated in the last 2 years of life (PRT{sub 2y}), and number of courses of PRT per thousand cancer deaths were measured in the benchmark population. These benchmarks were standardized to the characteristics of the overall population, and province-wide PRT rates were then compared to benchmarks. Results: Cases diagnosed at hospitals with no RT on-site and residents of poorer communities and those who lived farther from an RT center, were significantly less likely than others to receive PRT. However, availability of RT at the diagnosing hospital was the dominant factor. Neither socioeconomic status nor distance from home to nearest RT center had a significant effect on the use of PRT in patients diagnosed at a hospital with RT facilities. The benchmark population therefore consisted of patients diagnosed at a hospital with RT facilities. The standardized benchmark for PRT{sub lifetime} was 33.9%, and the corresponding province-wide rate was 28.5%. The standardized benchmark for PRT{sub 2y} was 32.4%, and the corresponding province-wide rate was 27.0%. The standardized benchmark for the number of courses of PRT per thousand cancer deaths was 652, and the corresponding province-wide rate was 542. Conclusions: Approximately one-third of patients who die of cancer in Ontario need PRT, but many of them are never treated.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

DOE OSTI2014

Benchmark Evaluation of HTR-PROTEUS Pebble Bed Experimental Program

Benchmark models were developed to evaluate 11 critical core configurations of the HTR-PROTEUS pebble bed experimental program. Various additional reactor physics measurements were performed as part of this program; currently only a total of 37 absorber rod worth measurements have been evaluated as acceptable benchmark experiments for Cores 4, 9, and 10. Dominant uncertainties in the experimental keff for all core configurations come from uncertainties in the ²³⁵U enrichment of the fuel, impurities in the moderator pebbles, and the density and impurity content of the radial reflector. Calculations of k eff with MCNP5 and ENDF/B-VII.0 neutron nuclear data are greater than the benchmark values but within 1% and also within the 3σ uncertainty, except for Core 4, which is the only randomly packed pebble configuration. Repeated calculations of k eff with MCNP6.1 and ENDF/B-VII.1 are lower than the benchmark values and within 1% (~3σ) except for Cores 5 and 9, which calculate lower than the benchmark eigenvalues within 4σ. The primary difference between the two nuclear data libraries is the adjustment of the absorption cross section of graphite. Simulations of the absorber rod worth measurements are within 3σ of the benchmark experiment values. The complete benchmark evaluation details are available in the 2014 edition of the International Handbook of Evaluated Reactor Physics Benchmark Experiments.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2020

Gaussian Process Optimization of Sensitivity-Based Similarity Metrics between New Nuclear Applications and New/Existing Benchmarks [Slides]

This presentation discusses Nuclear Criticality Safety (NCS) and how designing safe, new nuclear criticality experiments requires expert judgement, which could take years of experience. Sensitivity/uncertainty (S/U) analysis can be utilized by less experienced individuals to conservatively estimate uncertainties in important parameters, such as k eff , in newly proposed nuclear experiments. The presentation poses the question of how this analysis can be performed and states that the answer lies in matching new nuclear experiments with existing benchmark experiments using similarity metrics. By increasing the criticality safety of the application in this work, higher mass limits could be used in PF-4 operations. Additionally, the presentation discusses MCNP6.2®, Whisper-1.1, the software that can be used in this analysis. Also discussed is the fact that International Criticality Safety Benchmark Evaluation Project (ICSBEP) benchmarks rarely match new nuclear applications and that there are significant differences in given set of materials and/or geometry. If there are no benchmarks that match the application, the presentation discusses the possibility of creating new benchmarks. In conclusion, this work presents a Gaussian process (GP) optimization scheme that was used to generate new benchmarks with the highest sensitivity-based similarity metrics to user-defined nuclear applications. The Gaussian process optimization successfully designed 3 new experimental benchmarks that were highly correlated to the application of interest and had k eff values near critical. Optimization over c k,i-r has shown that investigating specific isotope reactions for different applications is crucial to designing benchmark experiments. Partial contribution from Pu dominates c k similarity metric. Future work includes testing new stand-alone similarity metrics or new combinations of similarity metrics as the design criterion of this optimization – design criterion is application dependent.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2023

EUI Benchmarks for Net-Zero Energy Buildings in India

In our study we present EUI benchmarks for NZEBs for six building types across residential and non-residential typologies and for India's five climate zones. This approach is similar to the simulation-based benchmarks used by the B3 program in Minnesota, the Cal-Arch methodology in California, and the US Solar Decathlon approach, which combine simulations with actual building data. Of the six building types we explore one in detail with a range of operation scenarios, specifically narrowing down the EUI benchmarks for mixed-mode building operation and for variable temperature set-points as prescribed in the National Building Code of India.The contribution of this work is to provide rigorous end-use level EUI benchmarks for six building types, and to describe a method for simulation-based EUI benchmarks for mixed-mode operation with variable setpoints to highlight the difference between the standard approach used for the five building types and the On-Site Construction Worker Housing which additionally has the mixed-mode variable setpoint approach.On-Site Construction Worker Housings are typically poorly constructed temporary structures, without adequate thermal comfort, It is critical to provide adequate thermal comfort to protect people from the warming effects of climate change, and to discover super efficient and cost-effective ways to do so.The methodology used for the On-Site Construction Worker Housing (CWH) results in an 80% acceptability according to the India Model for Adaptive Comfort in the National Building Code of India. EUIs of all six buildings are 60% lower than the minimum compliance with India's Energy Conservation Building Codes, providing benchmarks for efficiency levels. The end-use level EUI benchmarks are now provided to over 1800 Solar Decathlon India participants so that they can compare the performance of their NZEB designs.In particular, the CWH results provide an insight into the importance of the mixed-mode operation with variable temperature set-points. The results from the simulation study show that for an NZEB target, the EUI with standard thermal comfort model and without mixed operation is 58.26 kWh/m2*year, while that with the variable set-points of the adaptive model with mixed mode operation is 24.23 kWh/m2*year. This is a 58% reduction in EUI. Given that many building types including residential, and non-residential operate in mixed mode, it is important to take this work further to develop mixed-mode operation NZEB benchmarks so that the carbon intensity of these buildings could be lower than the standard thermal comfort model approach.

benchmarking↗

DOE OSTI2023

Opportunities for enhancing MLCommons efforts while leveraging insights from educational MLCommons earthquake benchmarks efforts

MLCommons is an effort to develop and improve the artificial intelligence (AI) ecosystem through benchmarks, public data sets, and research. It consists of members from start-ups, leading companies, academics, and non-profits from around the world. The goal is to make machine learning better for everyone. In order to increase participation by others, educational institutions provide valuable opportunities for engagement. In this article, we identify numerous insights obtained from different viewpoints as part of efforts to utilize high-performance computing (HPC) big data systems in existing education while developing and conducting science benchmarks for earthquake prediction. As this activity was conducted across multiple educational efforts, we project if and how it is possible to make such efforts available on a wider scale. This includes the integration of sophisticated benchmarks into courses and research activities at universities, exposing the students and researchers to topics that are otherwise typically not sufficiently covered in current course curricula as we witnessed from our practical experience across multiple organizations. As such, we have outlined the many lessons we learned throughout these efforts, culminating in the need for benchmark carpentry for scientists using advanced computational resources. The article also presents the analysis of an earthquake prediction code benchmark while focusing on the accuracy of the results and not only on the runtime; notedly, this benchmark was created as a result of our lessons learned. Energy traces were produced throughout these benchmarks, which are vital to analyzing the power expenditure within HPC environments. Additionally, one of the insights is that in the short time of the project with limited student availability, the activity was only possible by utilizing a benchmark runtime pipeline while developing and using software to generate jobs from the permutation of hyperparameters automatically. It integrates a templated job management framework for executing tasks and experiments based on hyperparameters while leveraging hybrid compute resources available at different institutions. The software is part of a collection called cloudmesh with its newly developed components, cloudmesh-ee (experiment executor) and cloudmesh-cc (compute coordinator).

58 GEOSCIENCES↗

NASA NTRS2022

A Benchmark Example for Delamination Propagation Predictions Based on the Calibrated End-Loaded Split Specimen

A benchmark example based on the Calibrated End-Loaded Split (C-ELS) specimen is developed and used to assess the performance of recently developed delamination propagation capabilities in the Abaqus/Standard finite element code. The C-ELS specimen has the advantage of a longer region of stable delamination propagation compared to the existing mode II benchmark case. The new benchmark example may therefore provide a better assessment tool by enabling more stable crack growth in regions further away from the boundary conditions or load application. First, a benchmark result is created manually using two-dimensional finite element models of the C-ELS specimen with different delamination lengths. Second, the performance of the delamination propagation capabilities in the Abaqus/Standard finite element code are assessed by comparing the results to the benchmark case. Two examples with different starter delamination lengths are studied. A shorter starter length is chosen to create a scenario with unstable delamination propagation. A longer delamination causes stable delamination propagation. Detailed results from three-dimensional analyses with aligned and mis-aligned meshes and two levels of mesh refinement are provided for several permutations of numerical input parameters. In general, good agreement can be achieved between the results obtained from the quasi-static propagation analysis and the benchmark analysis. However, particular non-default settings are found to be most reliable, accurate, and numerically efficient. Numerical artifacts including anomalous unreleased nodes in the crack wake and zig-zag crack fronts occur, and further development of the Abaqus/Standard VCCT propagation may be required. Use of the benchmark case to assess the continuous improvements in one finite element code illustrates the value of establishing benchmark solutions.

delamination↗

DOE OSTI2020

Availability of Shielding Benchmark Experiment Data in the ICSBEP Handbook

The International Criticality Safety Benchmark Evaluation Projects (ICSBEP) has over two decades of experience under the auspices of the Organisation for Economic Co-operation and Development (OECD) Nuclear Energy Agency (NEA) to identify, evaluate, and compile a comprehensive data set of criticality safety benchmark data [1]. The activities of the ICSBEP have led to the continued success of the International Handbook of Evaluated Criticality Safety Benchmark Experiments (ICSBEP Handbook) [2]. The contents of the ICSBEP Handbook include benchmark specifications derived from nuclear facilities around the world, which have been utilized to support criticality safety efforts and also integral testing of nuclear data [3]. The 2020 edition of the ICSBEP Handbook will include data from 582 evaluations containing benchmark specifications for 5,053 critical, subcritical, or near-critical configurations. There are seven criticality-alarm-placement/shielding evaluations containing a total of 45 benchmark configurations, and ten fundamental physics evaluations containing a total of 237 measurements relevant to criticality safety applications. The purpose of this paper is to summarize those benchmarks relative to validation needs in shielding applications currently found within the ICSBEP Handbook. The current and future plans of shielding benchmarks with regards to the ICSBEP will also be discussed.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗