Search NASA⌕ Search

SEARCH · Search NASA

Results for “performance benchmarking”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 541 records · Page 30

An HPC benchmark survey and taxonomy for characterization

The field of High-Performance Computing (HPC) is defined by providing computing devices with highest performance for a variety of demanding scientific users. The tight co-design relationship between HPC providers and users propels the field forward, paired with technological improvements, achieving continuously higher performance and resource utilization. A key device for system architects, architecture researchers, and scientific users are benchmarks, allowing for well-defined assessment of hardware, software, and algorithms. Many benchmarks exist in the community, from individual niche benchmarks testing specific features, to large-scale benchmark suites for whole procurements. We survey the available HPC benchmarks, summarizing them in table form with key details and concise categorization, also through an interactive website. For categorization, we present a benchmark taxonomy for well-defined characterization of benchmarks.

Benchmarking↗

OptiBench: An Optimization Benchmark Tool for Renewable Energy Problems

We propose a benchmark framework and visualization tool, OptiBench, for analyzing the performance of state-of-the-art optimization solvers across a variety of optimization problems in renewable energy research. Our framework is designed from the ground up in the Julia programming language and enables analysis at scale on high performance computing (HPC) systems. Our visualization tool allows effortless evaluation of optimization solver performance, robustness, and accuracy through intuitive plots, e.g., performance profiles, heat maps, and distribution plots. We have tested three benchmark suites relevant to the modeling of renewable energy systems, viz., CUTEst, PGLib-OPF, and WaterTAP water treatment optimization problems. We illustrate benchmarking of CUTEst using OptiBench on the National Renewable Energy Laboratory's (NREL) HPC Kestrel. Our findings indicate that MA57 HSL linear solver demonstrated the best overall performance for an experimental IPOPT implementation. Our work is ongoing and we intend to add support for more optimization solvers and benchmark test suites in the future.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Implementing and Benchmarking the Locally Competitive Algorithm on the Loihi 2 Neuromorphic Processor

Neuromorphic processors have garnered considerable interest in recent years for their potential in enabling energy-efficient and high-speed computing. The Locally Competative Algorithm (LCA) has been utilized for power efficient sparse coding on neuromophic processors, including the first Loihi processor \cite{appletospikes, loihi1}. With the Loihi 2 processor enabling custom neuron models and graded spike communication, more complex implementations of LCA are possible \cite{loihi2}. We present a new implementation of LCA designed for the Loihi 2 processor and perform an initial set of benchmarks comparing it to LCA on CPU and GPU devices. In these experiments LCA on Loihi 2 is faster and orders of magnitude more efficient, while maintaining similar reconstruction quality. We find this performance improvement increases as the LCA parameters are tuned towards greater representation sparsity. Our study highlights the potential of neuromorphic processors, particularly Loihi 2, in enabling intelligent,autonomous, real-time processing on small robots, satellite where there are strict SWaP (small, lightweighr, and low-power) requirement. By demonstrating the superior performance of LCA on Loihi 2 compared to conventional computing device, our study suggests that Loihi 2 could be a valuable tool in advancing these types of applications. Overall, our study highlights the potential of neuromorphic processors for efficient and accurate data processing on resource-constrained devices.

Parpart, Gavin G.↗

OptiBench: An Optimization Benchmark Tool for Renewable Energy Problems

We propose a benchmark framework and visualization tool, OptiBench, for analyzing the performance of state-of-the-art optimization solvers across a variety of optimization problems in renewable energy research. Our framework is designed from the ground up in the Julia programming language and enables analysis at scale on high performance computing (HPC) systems. Our visualization tool allows effortless evaluation of optimization solver performance, robustness, and accuracy through intuitive plots, e.g., performance profiles, heat maps, and distribution plots. We have tested three benchmark suites relevant to the modeling of renewable energy systems, viz., CUTEst, PGLib-OPF, and WaterTAP water treatment optimization problems. We illustrate benchmarking of CUTEst using OptiBench on the National Laboratory of the Rockies's (NLR) HPC Kestrel. Our findings indicate that MA57 HSL linear solver demonstrated the best overall performance for an experimental IPOPT implementation. Our work is ongoing and we intend to add support for more optimization solvers and benchmark test suites in the future.

97 MATHEMATICS AND COMPUTING↗

So you think you can track?

This work introduces a multi-camera tracking dataset consisting of 234 hours of video data recorded concurrently from 234 overlapping HD cameras covering a 4.2 mile stretch of 8-10 lane interstate highway near Nashville, TN. Video is recorded in cooperation with Tennessee State Department of Transportation and its policies. The video is recorded during a period of high traffic density with 500+ objects typically visible within the scene and typical object longevities of 3-15 minutes. GPS trajectories from 270 vehicle passes through the scene are manually corrected in the video data to provide a set of ground-truth trajectories for recall-oriented tracking metrics, and object detections are provided for each camera in the scene (159 million total before cross-camera fusion). Initial benchmarking of tracking-by-detection algorithms is performed against the GPS trajectories, and a best HOTA of only 9.5% is obtained (best recall 75.9% at IOU 0.1, 47.9 average IDs per ground truth object), indicating the benchmarked trackers do not perform sufficiently well at the long temporal and spatial durations required for traffic scene understanding. Video data, scene information, and vehicle trajectories are made publicly available at i24motion.org.

42 ENGINEERING↗

Bragg Spot Finder (BSF): a new machine-learning-aided approach to deal with spot finding for rapidly filtering diffraction pattern images

Macromolecular crystallography contributes significantly to understanding diseases and, more importantly, how to treat them by providing atomic resolution 3D structures of proteins. This is achieved by collecting X-ray diffraction images of protein crystals from important biological pathways. Spotfinders are used to detect the presence of crystals with usable data, and the spots from such crystals are the primary data used to solve the relevant structures. Having fast and accurate spot finding is essential, but recent advances in synchrotron beamlines used to generate X-ray diffraction images have brought us to the limits of what the best existing spotfinders can do. This bottleneck must be removed so spotfinder software can keep pace with the X-ray beamline hardware improvements and be able to see the weak or diffuse spots required to solve the most challenging problems encountered when working with diffraction images. In this paper, we first present Bragg Spot Detection (BSD), a large benchmark Bragg spot image dataset that contains 304 images with more than 66 000 spots. We then discuss the open source extensible U-Net-based spotfinder Bragg Spot Finder (BSF), with image pre-processing, a U-Net segmentation backbone, and post-processing that includes artifact removal and watershed segmentation. Finally, we perform experiments on the BSD benchmark and obtain results that are (in terms of accuracy) comparable to or better than those obtained with two popular spotfinder software packages ( Dozor and DIALS ), demonstrating that this is an appropriate framework to support future extensions and improvements.

36 MATERIALS SCIENCE↗

A comparative analysis of YOLOv8 and U-Net image segmentation approaches for transmission electron micrographs of polycrystalline thin films

Metallic thin films offer a platform to experimentally study the dynamics of microstructural evolution, but the required transmission electron microscopy (TEM)-based imaging generates complex images that are challenging to segment and quantify. This work provides a comparative analysis of a new YOLOv8 model and an established U-Net model for bright-field TEM images of polycrystals, employing a framework leveraging physical observables to evaluate performance against two hand-traced benchmark datasets. This methodology obviates the comparison of large, diversely structured, and manually labeled datasets that are required to assess performance on a per-image/per-pixel basis. It is found that the YOLOv8 model, adapted for real-time instance segmentation, has up to 43× faster inferencing (NVIDIA GeForce RTX 4090) compared to U-Net and reconstructs hand-traced grain size distributions (GSDs) with excellent fidelity, finding mean diameter within 3% for grains near an optimal magnification; for grains that deviate from the optimal pixel-diameter, the size of small- (large)-diameter grains is systematically over- (under)-estimated. This is partially mitigated by including scale-aware augmentations during training. Moreover, when the bias is corrected post-inference by a rigid shift in distribution, the YOLOv8 model reproduces ground truth GSDs with exceptional fidelity, with statistical tests indicating <5% probability that the distributions are distinct. Based on ground truth data, calibration curves pertaining to this shift can be constructed for a given model. This issue is not present in the U-Net model’s results, indicating that for quantitative measurements where the true size of objects is of interest, special procedures must be implemented for YOLO-based models.

36 MATERIALS SCIENCE↗

Qubit Assignment Using Time Reversal

As quantum computers with large numbers of qubits become increasingly available, experiments executed on a given device may not utilize all available qubits. In this case, the outcome of executing a quantum program will depend on the ability to efficiently select a subset of high-performing physical qubits. For any given quantum program and device there are many ways to assign physical qubits for execution of the program, and assignments will differ in performance due to the variability in quality across qubits and entangling operations on a single device. Evaluating the performance of each assignment using fidelity estimation introduces significant experimental overhead and will be infeasible for many applications, while relying on standard device benchmarks provides incomplete information about the performance of any specific program. Furthermore, the number of possible assignments grows combinatorially in the number of qubits on the device and in the program, motivating the use of heuristic optimization techniques. We demonstrate a practical solution to the problem of qubit assignment by using simulated annealing with a cost function based on the Loschmidt echo, a diagnostic that measures the reversibility of a quantum process. We provide theoretical justification for this choice of cost function by demonstrating that the optimal qubit assignment coincides with the optimal qubit assignment based on state fidelity in the weak error limit, and we provide experimental justification using diagnostics performed on Google’s superconducting qubit devices. We then establish the performance of simulated annealing for qubit assignment using classical simulations of noisy devices as well as optimization experiments performed on a quantum processor. Our results demonstrate that the use of Loschmidt echoes and simulated annealing provides a scalable and flexible approach to optimizing qubit assignment on near-term hardware.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Robust implementation of generative modeling with parametrized quantum circuits

Although the performance of hybrid quantum-classical algorithms is highly dependent on the selection of the classical optimizer and the circuit ansätze, a robust and thorough assessment on-hardware of such features has been missing to date. From the optimizer perspective, the primary challenge lies in the solver’s stochastic nature, and their significant variance over the random initialization. Therefore, a robust comparison requires one to perform several training curves for each solver before one can reach conclusions about their typical performance. Since each of the training curves requires the execution of thousands of quantum circuits in the quantum computer, such a robust study remained a steep challenge for most hybrid platforms available today. In this work, we leverage on Rigetti’s Quantum Cloud Services (QCS™) to overcome this implementation barrier, and we study the on-hardware performance of the data-driven quantum circuit learning (DDQCL) for three different state-of-the-art classical solvers, and on two-different circuit ansätze associated to different entangling connectivity graphs for the same task. Additionally, we assess the gains in performance from varying circuit depths. To evaluate the typical performance associated with each of these settings in this benchmark study, we use at least five independent runs of DDQCL towards the generation of quantum generative models capable of capturing the patterns of the canonical Bars and Stripes dataset. In this experimental benchmarking, the gradient-free optimization algorithms show an outstanding performance compared to the gradient-based solver. In particular, one of them had better performance when handling the unavoidable noisy objective function to be minimized under experimental conditions.

97 MATHEMATICS AND COMPUTING↗

Development and Validation of a Two-Phase Thermal-Hydraulic CFD Code NEK-2P

A project is underway to develop, verify and validate an advanced two-phase flow modeling capability for the highly-scalable, high-performance Computational Fluid Dynamics (CFD) code NEK5000. The goal of this work is to verify and validate the two-phase version of the NEK5000 code, named NEK-2P, to simulate the two-phase flow and heat transfer phenomena that occur in a Boiling Water Reactor (BWR) fuel bundle under various operating conditions. The NEK-2P two-phase flow models follow the approach used for the Extended Boiling Framework (EBF) previously developed at Argonne but include more fundamental physical models of boiling phenomena and advanced numerical algorithms for improved computational accuracy, robustness, and computational speed. The development of the NEK-2P two-phase solver and the implementation of the Extended Boiling Framework two-phase models were initially supported by Argonne National Laboratory (Argonne) through a Laboratory Directed Research and Development (LDRD) project during FY2014-2016. The development and validation of the two-phase models through analyses of selected two-phase boiling flow experiments was supported by the Nuclear Energy Advanced Modeling and Simulation (NEAMS) program in FY2017-2020. This report focuses on verification and validation of the water-steam boiling model NEK-2P Two-Phase, CFD code. The NEK-2P was validated with Nuclear Power Engineering Corporation (NUPEC) Pressurized Water Reactor (PWR) Sub-channel and Bundle Test (PSBT) void distribution benchmark. Three different simulations were performed and analyzed for various operating conditions such as wall-heat flux and sub-cooled inlet temperatures. Reasonably good agreement with measured data was obtained in predicting the measured void distributions. Simulations were performed for Virginia Tech. (VT) 3x3 rod bundle geometry with and without spacers. The preliminary results were presented for Simplified Spacer Grid (SSG). In addition, the implementation of interface reconstruction model was tested with one of the Becker benchmark Critical Heat Flux (CHF) experiments.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Polymer Hole Transport Material Functional Group Tuning for Improved Perovskite Solar Cell Performance

As lead halide perovskites (LHPs) continue to achieve success as a light-harvesting material in perovskite solar cells (PSCs), exploring and understanding other materials in the device stack become increasingly important. Particularly, selection of suitable hole transport materials (HTMs) that demonstrate high performance and stability is imperative in the design of P-I-N PSCs. Presented here are a family of 12 structurally related polymers based on either fluorene or carbazole main chains with select aromatic side groups that introduce tunable properties for use in PSCs. How properties such as the highest occupied molecular orbital energy level, conductivity, glass-transition temperature, and wettability of the HTM affect the PSC performance is explored. Devices that incorporate the polymer HTMs perform well relative to PTAA in benchmark P-I-N PSC architectures while exhibiting similar or superior stability under accelerated aging studies. The relative synthetic simplicity and resultant performance of the HTMs in PSCs coupled with the ability to customize properties with different functional groups demonstrates the potential of this family of HTMs for a variety of LHP materials.

14 SOLAR ENERGY↗

National Criticality Experiments Research Center (NCERC) - capabilities and recent measurements

The National Criticality Experiments Research Center (NCERC) located at the Device Assembly Facility (DAF) at the Nevada National Security Site (NNSS) and operated by Los Alamos National Laboratory (LANL) is home to four critical assemblies which are used to support of range of missions, including nuclear criticality safety and nuclear nonproliferation. Additionally, subcritical systems can also be assembled at NCERC. NCERC is providing critical and subcritical experiments valuable to the nuclear data community and experiments performed at NCERC are often published as benchmarks in the International Criticality Safety Benchmark Evaluation Project (ICSBEP) Handbook. This manuscript will give a broad overview of recent experiments performed at NCERC, upcoming experiments, and why integral measurements are important and useful to the nuclear data community. The four critical assemblies are GODIVA IV, FLATTOP, COMET, and PLANET. GODIVA IV is a cylindrical metal fast burst reactor, the fourth in the GODIVA series that dates back to the 1950’s. FLATTOP is an highly enriched uranium (HEU) or Pu core reflected by natural uranium. COMET and PLANET are vertical lift assemblies, where one half of the reactor can be lifted to the upper half of the reactor to create a critical system. Some recent experiments include various critical intermediate energy assemblies with lead, and subcritical measurements of plutonium reflected by copper, tungsten, and nickel. Work is also underway to make a better measurement of the critical mass of neptunium, using a neptunium sphere surrounded by nickel shells. Additionally, measurements will be performed next year with HEU shells from Rocky Flats. These HEU shells will be stacked together to make larger systems, allowing for a large range of criticality (from subcritical to delayed critical). Other upcoming measurements include an HEU critical assembly sensitive to intermediate energy neutrons.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

IRIS-MASH: Efficient Multi-device Asynchronous Multi-Stream Heterogeneous Computing

In the rapidly evolving field of high-performance computing (HPC), effectively leveraging heterogeneous devices through asynchronous task programming is paramount. This paper presents a robust asynchronous task programming model tailored for a multi-device, multi-stream execution environment that incorporates a diverse array of heterogeneous computing units, including GPUs from various vendors and other accelerators. Current state-of-the-art task programming models provide methodologies to support asynchronous task executions, but they typically handle homogeneous devices using native programming languages, while support for heterogeneous devices is limited to frameworks like OpenCL. This gap presents significant challenges in abstracting heterogeneous devices to harness their true asynchronous capabilities effectively using their native programming languages. By implementing asynchronous task execution, our model significantly boosts the performance of tiled algorithm task graphs through overlapping data transfers with computation and enabling the simultaneous execution of multiple kernels. We integrate this approach into a heterogeneous Intelligent Runtime System (IRIS) and assess its performance using a suite of tiled algorithm benchmarks from the heterogeneous math kernels library (MatRIS) based on IRIS. Experimental results demonstrate a performance improvement ranging from 1.6 × to 2 × over IRIS without asynchronous support, and a notable 22% performance enhancement compared to established runtime systems such as StarPU and PaRSEC. This approach significantly improves computation efficiency of HPC workflows and provides a solid base for future exploration and development in the area of asynchronous task programming in heterogeneous systems.

Miniskar, Narasinga Rao [ORNL] (ORCID:000000018259↗

Qualifying LaBr3:Ce+Sr detector performance for the Mu2e experiment at Fermilab Using the ELBE Accelerator

A LaBr3:Ce+Sr detector will be used to measure the stopped muon captures at the Mu2e experiment at Fermilab. It has been benchmarked in a test beam experiment performed at the ELBE electron accelerator located at the Helmholtz-Zentrum Dresden-Rossendorf, Germany. ELBE’s pulsed bremsstrahlung beam line was set to deliver an average γ -ray energy of between 4–5 MeV. The detector response was mapped to match Mu2e beam conditions, including rates up to 1 Mcps, energy flux, and time structure. A radioactive calibration source was used to mimic the characteristic 1808.7 keV γ -ray, emitted during the atomic muon nuclear capture in the Mu2e aluminum stopping target. The detector energy resolution was measured as a function of the average energy flux: up to 1 TeV/s for 0.34 s, the steady-operation beam-on time and up to 4 TeV/s for 5 ms to get a conservative estimate of the effect of high intensity Mu2e beam fluctuations. The PMT gain variation as a function of the beam spill length and average intensity has been parametrized and corrected for. When a PMT gain correction corresponding to the average beam-spill intensity is applied, the residual effect of beam intensity fluctuations around the average degrades the energy resolution, σ E γ /E γ , at 1808.7 keV from 0.66% to 0.83%.

Huang, Shihua [Purdue U., West Lafayette]↗

Neural Image Compression: Generalization, Robustness, and Spectral Biases

Recent advances in neural image compression (NIC) have resulted in models which are starting to outperform traditional codecs. While this has led to growing excitement about using these methods in real-world applications, the successful adoption of any machine learning system (including NIC) in the wild requires it to generalize (and be robust) to unseen distribution shifts at deployment time. Unfortunately, current research lacks comprehensive datasets and informative tools to evaluate and understand compression performance in real-world settings. To bridge this crucial gap, first, this paper presents a comprehensive benchmark suite to evaluate the out-of-distribution (OOD) performance of image compression methods. Specifically, we design CLIC-C and Kodak-C by introducing 15 common corruptions to popular CLIC and Kodak benchmarks. Next, we propose spectrally inspired introspection tools to gain a deeper understanding of errors introduced by image compression methods as well as their OOD performance. To this end, we carry out a detailed performance comparison of the classical codec with various variants of NIC (e.g., original, variable rate, pruned), revealing intriguing findings that challenge our current understanding of the strengths and limitations of NIC. Finally, we corroborate our empirical findings with theoretical analysis, providing an in-depth view of the OOD performance of NIC. Our benchmarks, spectral introspection tools, and findings provide a crucial bridge to the real-world adoption of NIC. We hope that our work will propel future efforts in designing more robust and generalizable NIC methods.

neural networks, Variational Autoencoder, robustne↗

Disjunctive linear separation conditions and mixed-integer formulations for aircraft conflict resolution

In this paper, we address the aircraft conflict resolution problem in air traffic control. We introduce new mixed-integer programming formulations for aircraft conflict resolution with speed, heading and altitude control which are based on disjunctive linear separation conditions. We first examine the two-dimensional aircraft conflict resolution problem with speed and heading control represented as continuous decision variables. We show that the proposed disjunctive linear separation conditions are equivalent to the classical nonlinear conditions for aircraft separation. Further, we characterise conflict-free trajectories based on aircraft velocity bounds and propose a simple pre-processing algorithm to identify aircraft pairs which are either always conflict-free, or which cannot be separated using speed and heading control only. We then incorporate altitude control and propose a lexicographic optimisation formulation that aims to minimise the number of flight level changes before resolving outstanding conflicts via two-dimensional velocity control. The proposed mixed-integer programming formulations are nonconvex, and we propose convex relaxations, decomposition methods and constraint generation algorithms to solve the two-dimensional and lexicographic optimisation formulations to guaranteed optimality. Numerical experiments on four types of conflict resolution benchmarking instances are conducted to test the performance of the proposed mixed-integer formulations. Further, the proposed method is compared against two benchmarks based on state-of-the-art approaches for the aircraft conflict resolution problem. Our numerical results show that the proposed method largely outperforms both benchmarks in terms of runtime and is able to solve significantly more instances to global optimality.

97 MATHEMATICS AND COMPUTING↗

CO 2 storage site characterization using ensemble-based approaches with deep generative models

Estimating spatially distributed properties such as permeability from available sparse measurements is a great challenge in efficient subsurface CO 2 storage operations. In this paper, a deep generative model that can accurately capture complex subsurface structure is tested with an ensemble-based inversion method for accurate and accelerated characterization of CO 2 storage sites. We chose Wasserstein Generative Adversarial Network with Gradient Penalty (WGAN-GP) for its realistic reservoir property representation and Ensemble Smoother with Multiple Data Assimilation (ES-MDA) for its robust data fitting and uncertainty quantification capability. WGAN-GP are trained to generate high-dimensional permeability fields from a low-dimensional latent space and ES-MDA then updates the latent variables by assimilating available measurements. Several subsurface site characterization examples including Gaussian, channelized, and fractured reservoirs are used to evaluate the accuracy and computational efficiency of the proposed method and the main features of the unknown permeability fields are characterized accurately with reliable uncertainty quantification. Furthermore, the estimation performance is compared with a widely-used variational, i.e., optimization-based, inversion approach, and the proposed approach outperforms the variational inversion method in several benchmark cases. We explain such superior performance by visualizing the objective function in the latent space: because of nonlinear and aggressive dimension reduction via generative modeling, the objective function surface becomes extremely complex while the ensemble approximation can smooth out the multi-modal surface during the minimization. This suggests that the ensemble-based approach works well over the variational approach when combined with deep generative models at the cost of forward model runs unless convergence-ensuring modifications are implemented in the variational inversion.

42 ENGINEERING↗

Biochemical Conversion of Lignocellulosic Biomass to Hydrocarbon Fuels and Products: 2020 State of Technology and Future Research

The annual State of Technology (SOT) assessment is an essential activity for biochemical platform research. It allows the impact of research progress to be quantified in terms of economic improvements in the overall cellulosic biofuel production process for a particular conversion pathway. As such, initial benchmarks can be established for currently demonstrated performance and progress can be tracked towards out-year goals to ultimately demonstrate cost-competitive cellulosic biofuel technology. The purpose of this report is to benchmark the latest experimental developments across a number of potential bioconversion pathways as quantified by modeled minimum fuel selling prices (MFSPs), as a measure of current status relative to those final targets. For this state of technology, TEA models were run for two separate biological conversion pathways to fuels, based on available data for integrated biomass deconstruction and hydrolysate processing; namely carboxylic acids (primarily butyric acid) and diols (2,3-butanediol [BDO]), reflecting NREL's recently-published 2018 biochemical design report focused on those two pathways. The models were run across three scenarios for lignin utilization, namely combustion, conversion to coproducts based on "base case" performance with biomass hydrolysate, and conversion to coproducts based on "high" performance demonstrated with model lignin monomer components. Relative to prior 2019 benchmarks, the 2020 SOT achieved further MFSP reductions of approximately $1/gallon gasoline equivalent, primarily attributed to improvements in pretreatment that led to both cost reductions and yield improvements.

09 BIOMASS FUELS↗