Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmarking Software”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

Systematic Benchmarking of Climate Models: Methodologies, Applications, and New Directions

As climate models become increasingly complex, there is a growing need to comprehensively and systematically assess model performance with respect to observations. Given the increasing number and diversity of climate model simulations in use, the community has moved beyond simple model intercomparison and toward developing methods capable of benchmarking a large number of simulations against a suite of climate metrics. Here, we present a detailed review of evaluation and benchmarking methods and approaches developed in the last decade, focusing primarily on scientific implications for Coupled Model Intercomparison Project (CMIP) simulations and CMIP6 results that contributed to the Intergovernmental Panel on Climate Change (IPCC) Sixth Assessment Report (AR6). Based on this review, we explain the resulting contemporary philosophy of model benchmarking, and provide clear distinctions and definitions of the terms model verification, process validation, evaluation, and benchmarking. While significant progress has been made in model development based on systematic evaluation and benchmarking efforts, some climate system biases still remain. The development of open‐source community software packages has played a fundamental role in identifying areas of significant model improvement and bias reduction. We review the key features of several software packages that have been commonly used over the past decade to evaluate and benchmark global and regional climate models. Additionally, we discuss best practices for the selection of evaluation and benchmarking metrics and for interpreting the obtained results, the importance of selecting suitable sources of reference data and accurate uncertainty quantification.

Environmental sciences↗

Near real-time streaming analysis of big fusion data

Experiments on fusion plasmas produce high-dimensional data time series with ever-increasing magnitude and velocity, but turn-around times for analysis of this data have not kept up. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article, we introduce the Delta framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a high-performance compute center, Delta allows computationally expensive data analysis tasks to be performed in between plasma pulses. This article describes the modular and expandable software architecture of Delta and presents performance benchmarks of individual components as well as of an example workflow. Focusing on a streaming analysis workflow where electron cyclotron emission imaging (ECEi) data is measured at KSTAR on the National Energy Research Scientific Computing Center's (NERSC's) supercomputer we routinely observe data transfer rates of about 4 Gigabit per second. In NERSC, a demanding turbulence analysis workflow effectively utilizes multiple nodes and graphical processing units and executes them in under 5 min. We further discuss how Delta uses modern database systems and container orchestration services to provide web-based real-time data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. Here, by providing session leaders and physics operators, results of higher-order data analysis using live visualizations may make more informed decisions on how to configure the machine for the next shot.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Fast intensity-modulated arc therapy based on 2-step beam segmentation

Purpose: Single or few arc intensity-modulated arc therapy (IMAT) is intended to be a time saving irradiation method, potentially replacing classical intensity-modulated radiotherapy (IMRT). The aim of this work was to evaluate the quality of different IMAT methods with the potential of fast delivery, which also has the possibility of adapting to the daily shape of the target volume. Methods: A planning study was performed. Novel double and triple IMAT techniques based on the geometrical analysis of the target organ at risk geometry (2-step IMAT) were evaluated. They were compared to step and shoot IMRT reference plans generated using direct machine parameter optimization (DMPO). Volumetric arc (VMAT) plans from commercial preclinical software (SMARTARC) were used as an additional benchmark to classify the quality of the novel techniques. Four cases with concave planning target volumes (PTV) with one dominating organ at risk (OAR), viz., the PTV/OAR combination of the ESTRO Quasimodo phantom, breast/lung, spine metastasis/spinal cord, and prostate/rectum, were used for the study. The composite objective value (COV) and other parameters representing the plan quality were studied. Results: The novel 2-step IMAT techniques with geometry based segment definition were as good as or better than DMPO and were superior to the SMARTARC VMAT techniques. For the spine metastasis, the quality measured by the COV differed only by 3%, whereas the COV of the 2-step IMAT for the other three cases decreased by a factor of 1.4-2.4 with respect to the reference plans. Conclusions: Rotational techniques based on geometrical analysis of the optimization problem (2-step IMAT) provide similar or better plan quality than DMPO or the research version of SMARTARC VMAT variants. The results justify pursuing the goal of fast IMAT adaptation based on 2-step IMAT techniques.

60 APPLIED LIFE SCIENCES↗

NISQ Benchmarking

Test suite of quantum algorithms for Noisy Intermediate Scale Quantum (NISQ) computers. The test suite includes benchmark-style code for quantum volume circuits (QV), fairness sampling circuits, quantum telecloning circuits, and other NISQ benchmark style algorithms on small problems (i.e., up to 100 qubits), such as Variational Quantum Eigensolver (VQE), Hamiltonian Simulation, and Grover unstructured search example circuits. These benchmark-style applications are implemented in quantum software packages, mostly IBM's QISKIT, but may include vendor-specific frameworks, such as PyQuil (for Rigetti) or Q\# for Microsoft, or CirQ (for Google) as the test suite grows with the vendor sample. The test suite also includes numerical simulation code for Quantum Alternating Operator Ansatz (QAOA) algorithms, VQE, Hamiltonian Simulation and search examples. Numerical simulation code simulates quantum computers on classical computers, which is only possible for small problem instances; the implementation framework of choice is typically within Python, using the numpy/scipy libraries as well as extensions to the Julia language.

Pelofske, Elijah↗

Using SciDB to Support Photon Science Data Analysis

Array data analytic systems like SciDB hold great potential to accelerate processing data from SLAC's Linac Coherent Light Source and other experiments. SciDB is unique in its ability to integrate storage and processing of array data efficiently, providing both space-efficient storage and out-of-memory efficient parallel array processing. We describe a recent effort to leverage SciDB to store and process LCLS data. The work includes development of software to import data into SciDB, subsequent benchmarks, and interactive manipulation of data in SciDB.

43 PARTICLE ACCELERATORS↗

Stand Alone Battery Thermal Management System

The objective of this project is research, development and demonstration of innovative thermal management concepts that reduce the cell or battery weight, complexity (component count) and/or cost by at least 20%. The project addresses two issues that are common problems with current state of the art lithium ion battery packs used in vehicles; low power at cold temperatures and reduced battery life when exposed to high temperatures. Typically, battery packs are “oversized” to satisfy the two issues mentioned above. The first phase of the project was spent making a battery pack simulation model using AMEsim software. The battery pack used as a benchmark was from the Fiat 500EV. FCA and NREL provided vehicle data and cell data that allowed an accurate model to be created that matched the electrical and thermal characteristics of the actual battery pack. The second phase involved using the battery model from the first phase and evaluate different thermal management concepts. In the end, a gas injection heat pump system was chosen as the dedicated thermal system to both heat and cool the battery pack. Based on the simulation model. The heat pump system could use 50% less energy to heat the battery pack in -20°C ambient conditions, and by keeping the battery cooler at hot climates, the battery pack size could be reduced by 5% and still meet the warranty requirements. During the final phase, the actual battery pack and heat pump system were installed in a test bench at DENSO to validate the simulation results. Also during this phase, the system was moved to NREL where testing was also done to validate the results. In conclusion, the heat pump system can improve “fuel economy” (for electric vehicle) by 12% average in cold climates. Also, the battery pack size, or capacity, could be reduced 5%, or if pack size is kept constant, the pack life could be increased by two years. Finally, the total battery pack and thermal system cost could be reduced 5% only if the system is integrated with the vehicle cabin air conditioning system. The reason why we were not able to achieve the 20% reduction target is because of the natural decay of the battery cell due to the number of cycles. Perhaps newer battery chemistries that are not so sensitive to cycling would have more potential for reducing the battery size due to thermal issues.

25 ENERGY STORAGE↗

Towards exact finite temperature electronic structure in solids and molecules (Final Technical Report)

This report describes the University of Iowa portion of a project that is now continuing at Michigan State University. We are developing novel methods, algorithms, and software to enable simulations of molecules and materials at high temperature. This is a key challenge in chemistry and materials science. By refining an approach called Density Matrix Quantum Monte Carlo (DMQMC), we developed faster and more accurate ways to conduct these simulations. These advances will help us understand how temperature affects the behavior of electrons, chemical bonds, and phase transitions in solids and molecules. These breakthroughs are especially important for applications where light and heat drive chemical reactions, superconductivity, and materials used in energy and sensing. In addition, this project involved the development of the open-source HANDE-QMC software package, supporting the broader community in benchmarking and developing finite-temperature electronic structure methods.

36 MATERIALS SCIENCE↗

Porosity Area Fraction Analysis of Bonded Borosilicate Specimens From in-Situ Optical Imagery

Image segmentation methods are routinely used to analyze data derived in nondestructive testing scenarios. A typical problem in this area is to estimate porosity from X-ray Computed Tomography (CT) inspections. This investigation aims to understand the effectiveness of open-source image segmentation methods for porosity estimation from optical images in the context of strength analysis of bonded materials and provides a benchmark of performance against a standard commercial software package. The rank ordering of the specimens by porosity area fraction is consistent across both methods.

Porosity area fraction↗

Porosity Area Fraction Analysis of Bonded Borosilicate Specimens From in-Situ Optical Imagery

Image segmentation methods are routinely used to analyze data derived in nondestructive testing scenarios. A typical problem in this area is to estimate porosity from X-ray Computed Tomography (CT) inspections. This investigation aims to understand the effectiveness of open-source image segmentation methods for porosity estimation from optical images in the context of strength analysis of bonded materials and provides a benchmark of performance against a standard commercial software package. The rank ordering of the specimens by porosity area fraction is consistent across both methods.

Porosity area fraction↗

U-10Mo Conversion Burnup Benchmarking Activities for Full-Size LEU Fuel Plates

Validation benchmark experiments for low-enriched uranium (LEU) U-Mo fuel systems are currently being developed from existing experiments irradiated in the Advanced Test Reactor. (ATR) and ATR Critical Facility (ATRC). The experiments chosen include the ATR Full size plate In center Flux trap Position (AFIP-7) experiment and the Ki-Jang Research Reactor Lead Test Assembly (KJRR-LTA) located in the Northeast Flux Trap. Each experiment was modeled using a high-fidelity three-dimensional Monte Carlo code with depletion capabilities. A description of the validation experiments will be provided as well as comparison to post irradiation examination results for the AFIP-7 experiment. The PIE results for the AFIP-7 experiment have demonstrated good agreement with analytical models. Due to the complex nature in of the ATR, modeling simplifications to support benchmark activities using lower fidelity reactor physics software. A discussion of the analytical bias associated with modeling simplifications will also be provided.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

SLIP (Surrogate Launching and Integration Platform)

SLIP (Surrogate Launching and Integration Platform) is a software ecosystem for downloading and running ML/AI benchmarks. SLIP automatically downloads and sets up the code and data running a benchmark. The large data sets and model code are cached locally with a specific ID within a designated cache directory.

Brown, Cade↗

A new fuel modeling capability, CTFFuel, with a case study on the fuel thermal conductivity degradation

Here, a new fuel modeling capability, CTFFuel, is developed from the subchannel code, CTF. This code is a standalone interface to the CTF fuel rod models, allowing for fuel rod simulations to be run independently from the fluid. This paper provides an overview of the code with a case study on the thermal conductivity degradation of LWR fuels to demonstrate its capabilities. The modeling of fuel thermal conductivity degradation in the code is improved through the addition of new modeling options to account for the irradiation effects via globally defined parameters. After the initial implementation, a variety of order-of-accuracy tests and code comparisons are performed to test software quality. A controlled analysis is allowed by CTFFuel to verify the numerical scheme of CTF’s conduction solution and to benchmark its fuel temperature predictions against FRAPCON-4.0’s. Overall, the software quality and verification procedure ensures that the new model is coded correctly, that it properly interacts with the rest of the code.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Benchmarking Algorithm for Radio Nuclide Identification (BARNI) Literature Review

This is a literature review of published radioisotope identification (ID) methods, and an attempt at systematic organization of the approaches for solving this problem. In addition, there is an overview of the related open and closed source software, and patents related to radioisotope identification. This document is also part of the development plan for BARNI (Benchmarking Algorithm for RadioNuclide Identification) software, since it is meant to be a comparison tool, a summary of radiation ID performance evaluations is also included.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Performance Portability Evaluation of OpenCL Benchmarks across Intel and NVIDIA Platforms

We evaluate the capabilities of vendor-provided OpenCL implementations for performance portability across multiple computing platforms. The Rodinia benchmark suite is used for this evaluation. We apply the metric defined by Pennycook et al., and we use roofline efficiency from the Roofline performance model as the "performance efficiency" in the metric's definition. We found that the delivered performance portability is similar for several benchmarks, even if the roofline-based performance efficiencies across platforms are very different among the benchmarks. To help distinguish between these instances, we extend the metric by adding the standard deviation of the performance efficiencies for each benchmark. We argue that the standard deviation gives additional insight into performance portability assessment since it adds the performance variability across platforms. Additionally, we discuss the challenges to measure performance portability associated with algorithms and system software. In terms of algorithms, we need to carefully construct the benchmarks and appropriately use the concurrency available on a platform. In terms of system software, we depend on the vendor performance tools to support the desired programming model and runtime to be able to measure the metrics of interest.

GPU↗

Experiment vs simulation RT WFNDEC 2014 benchmark: CIVA results

The French Atomic Energy Commission and Alternative Energies (CEA) has developed for years the CIVA software dedicated to simulation of NDE techniques such as Radiographic Testing (RT). RT modelling is achieved in CIVA using combination of a determinist approach based on ray tracing for transmission beam simulation and a Monte Carlo model for the scattered beam computation. Furthermore, CIVA includes various detectors models, in particular common x-ray films and a photostimulable phosphor plates. This communication presents the results obtained with the configurations proposed in the World Federation of NDEC 2014 RT modelling benchmark with the RT models implemented in the CIVA software.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Aircraft Engine Gas Path Diagnostic Methods: Public Benchmarking Results

Recent technology reviews have identified the need for objective assessments of aircraft engine health management (EHM) technologies. To help address this issue, a gas path diagnostic benchmark problem has been created and made publicly available. This software tool, referred to as the Propulsion Diagnostic Method Evaluation Strategy (ProDiMES), has been constructed based on feedback provided by the aircraft EHM community. It provides a standard benchmark problem enabling users to develop, evaluate and compare diagnostic methods. This paper will present an overview of ProDiMES along with a description of four gas path diagnostic methods developed and applied to the problem. These methods, which include analytical and empirical diagnostic techniques, will be described and associated blind-test-case metric results will be presented and compared. Lessons learned along with recommendations for improving the public benchmarking processes will also be presented and discussed.

Simon, Donald L.↗

Evaluation of China Experimental Fast Reactor Start-up Tests (Final Report, Revision 1)

This report documents the results and observations on the Coordinated Research Project (CRP) of the International Atomic Energy Agency (IAEA) on “Neutronics Benchmark of CEFR Start-Up Tests.” The China Experimental Fast Reactor (CEFR) is a 65MWt sodium-cooled fast reactor with highly enriched uranium oxide fuel. The reactor achieved first criticality in 2010, and a series of start-up tests were conducted to measure various reactor physics parameters. In 2018, the IAEA launched the CRP for validation and qualification of member states' computation capabilities in the field of fast reactor simulation by utilizing the measured data in the CEFR start-up test. Twenty-nine international organizations from eighteen member countries, including Argonne National Laboratory, have participated in the CRP. The CEFR start-up tests offer a rare opportunity to validate U.S. nuclear engineering software because it is a well-specified benchmark with corresponding measurements for a recently built fast reactor starting up with a known fuel composition (fresh fuel). Previous fast reactor validation benchmarks frequently involve reactors that have already been started up and contain irradiated fuel that is difficult to characterize with high certainty.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

TRINIDI (Time-of-Flight Resonance Imaging with Neutrons for Isotopic Density Inference)

This software is an open-source Python library that provides tools for processing hyperspectral neutron time-of-flight radiography data. This type of data allows material decomposed reconstructions to be generated with the use of material characteristic spectral responses and the algorithms provided in this code library. The software library will contain tools for pre-processing the neutron measurement data, estimating measurement system parameters, reconstructing material decomposed radiographs, and computing material decomposed computed tomography (CT). Furthermore, it will have capability to generate and process simulated neutron time-of-flight data with the goal of benchmarking and demonstrating the tools that are provided. The software will include thorough documentation and application examples.

Balke, Thilo↗