Search NASA⌕ Search

SEARCH · Search NASA

Results for “computer system benchmarking”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14

OptiBench: An Optimization Benchmark Tool for Renewable Energy Problems

We propose a benchmark framework and visualization tool, OptiBench, for analyzing the performance of state-of-the-art optimization solvers across a variety of optimization problems in renewable energy research. Our framework is designed from the ground up in the Julia programming language and enables analysis at scale on high performance computing (HPC) systems. Our visualization tool allows effortless evaluation of optimization solver performance, robustness, and accuracy through intuitive plots, e.g., performance profiles, heat maps, and distribution plots. We have tested three benchmark suites relevant to the modeling of renewable energy systems, viz., CUTEst, PGLib-OPF, and WaterTAP water treatment optimization problems. We illustrate benchmarking of CUTEst using OptiBench on the National Laboratory of the Rockies's (NLR) HPC Kestrel. Our findings indicate that MA57 HSL linear solver demonstrated the best overall performance for an experimental IPOPT implementation. Our work is ongoing and we intend to add support for more optimization solvers and benchmark test suites in the future.

97 MATHEMATICS AND COMPUTING↗

CI/CD Efforts for Validation, Verification and Benchmarking OpenMP Implementations

Software developers must adapt to keep up with the changing capabilities of platforms so that they can utilize the power of High-Performance Computers (HPC), including exascale systems. OpenMP, a directive-based parallel programming model, allows developers to include directives to existing C, C++, or Fortran code to allow node level parallelism without compromising performance. This paper describes our CI/CD efforts to provide easy evaluation of the support of OpenMP across different compilers using existing testsuites and benchmark suites on HPC platforms. Our main contributions include (1) the set of a Continuous Integration (CI) and Continuous Development (CD) workflow that captures bugs and provides faster feedback to compiler developers, (2) an evaluation of OpenMP (offloading) implementations supported by AMD, HPE, GNU, LLVM, and Intel, and (3) evaluation of the quality of compilers across different heterogeneous HPC platforms. With the comprehensive testing through the CI/CD workflow, we aim to provide a comprehensive understanding of the current state of OpenMP (offloading) support in different compilers and heterogeneous platforms consisting of CPUs and GPUs from NVIDIA, AMD, and Intel.

Jarmusch, Aaron↗

OptiBench: An Optimization Benchmark Tool for Renewable Energy Problems

We propose a benchmark framework and visualization tool, OptiBench, for analyzing the performance of state-of-the-art optimization solvers across a variety of optimization problems in renewable energy research. Our framework is designed from the ground up in the Julia programming language and enables analysis at scale on high performance computing (HPC) systems. Our visualization tool allows effortless evaluation of optimization solver performance, robustness, and accuracy through intuitive plots, e.g., performance profiles, heat maps, and distribution plots. We have tested three benchmark suites relevant to the modeling of renewable energy systems, viz., CUTEst, PGLib-OPF, and WaterTAP water treatment optimization problems. We illustrate benchmarking of CUTEst using OptiBench on the National Renewable Energy Laboratory's (NREL) HPC Kestrel. Our findings indicate that MA57 HSL linear solver demonstrated the best overall performance for an experimental IPOPT implementation. Our work is ongoing and we intend to add support for more optimization solvers and benchmark test suites in the future.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Adaptive variational quantum minimally entangled typical thermal states for finite temperature simulations

Scalable quantum algorithms for the simulation of quantum many-body systems in thermal equilibrium are important for predicting properties of quantum matter at finite temperatures. Here we describe and benchmark a quantum computing version of the minimally entangled typical thermal states (METTS) algorithm for which we adopt an adaptive variational approach to perform the required quantum imaginary time evolution. The algorithm, which we name AVQMETTS, dynamically generates compact and problem-specific quantum circuits, which are suitable for noisy intermediate-scale quantum (NISQ) hardware. We benchmark AVQMETTS on statevector simulators and perform thermal energy calculations of integrable and nonintegrable quantum spin models in one and two dimensions and demonstrate an approximately linear system-size scaling of the circuit complexity. We further map out the finite-temperature phase transition line of the two-dimensional transverse field Ising model. Finally, we study the impact of noise on AVQMETTS calculations using a phenomenological noise model.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Parallelization of NAS Benchmarks for Shared Memory Multiprocessors

This paper presents our experiences of parallelizing the sequential implementation of NAS benchmarks using compiler directives on SGI Origin2000 distributed shared memory (DSM) system. Porting existing applications to new high performance parallel and distributed computing platforms is a challenging task. Ideally, a user develops a sequential version of the application, leaving the task of porting to new generations of high performance computing systems to parallelization tools and compilers. Due to the simplicity of programming shared-memory multiprocessors, compiler developers have provided various facilities to allow the users to exploit parallelism. Native compilers on SGI Origin2000 support multiprocessing directives to allow users to exploit loop-level parallelism in their programs. Additionally, supporting tools can accomplish this process automatically and present the results of parallelization to the users. We experimented with these compiler directives and supporting tools by parallelizing sequential implementation of NAS benchmarks. Results reported in this paper indicate that with minimal effort, the performance gain is comparable with the hand-parallelized, carefully optimized, message-passing implementations of the same benchmarks.

Waheed, Abdul↗

Ab initio ground states of strongly-correlated materials on quantum computers

The accurate first-principles description of strongly-correlated materials is an important and challenging problem in condensed matter physics. Ab initio downfolding has emerged as a way of deriving accurate many-body Hamiltonians including strong correlations, representing a subspace of interest of a material, using density functional theory calculations as a starting point. However, the solution of these material-specific models can scale exponentially on classical computers, constituting a challenge. Here we propose that utilizing quantum computers for obtaining the properties of downfolded Hamiltonians yields an accurate description of the ground state properties of strongly-correlated systems, while circumventing the exponential scaling problem. We benchmark the solution of Hubbard-like models obtained through downfolding by utilizing a classical tensor network implementation of variational quantum eigensolvers (VQE), and we reveal a strategy for driving the optimization through a hybrid minimization of the energy and maximization of the overlap with an approximate solution obtained through low-cost computational methods. This results in a reduction of the energy error by orders of magnitude compared to conventional VQE approaches, and allows us to reproduce long-range correlations for the first time. We demonstrate our first-principles approach for diverse strongly-correlated materials, correctly predicting the antiferromagnetic state of one-dimensional cuprate Ca 2 CuO 3 , the excitonic ground state of monolayer WTe2, and the charge-ordered state of correlated metal SrVO 3 . Our efficient computational implementation allows us to simulate large systems with up to 54 qubits and encompassing up to four correlated bands, which is indicative of the complexity that our framework can address.

Antonios M Alvertis↗

A step towards the final frontier: Lessons learned from acceptance testing of the first HPE/Cray EX 3000 system at ORNL

Summary In this article, we summarize the deployment of the Air Force Weather (AFW) HPC11 system at Oak Ridge National Laboratory (ORNL) including the process followed to successfully complete acceptance testing of the system. HPC11 is the first HPE/Cray EX 3000 system that has been successfully released to its user community in a federal facility. HPC11 consists of two identical 800‐node supercomputers, Fawbush and Miller, with access to two independent and identical lustre parallel file systems. HPC11 is equipped with Slingshot 10 interconnect technology and relies on the HPE Performance Cluster Manager software for system configuration. ORNL has a clearly defined acceptance testing process used to ensure that every new system deployed can provide the necessary capabilities to support user workloads. We worked closely with HPE and AFW to develop a set of tests that used the United Kingdom's Meteorological Office's Unified Model and 4‐dimensional variational data assimilation. We also included benchmarks and applications from the Oak Ridge Leadership Computing Facility portfolio to fully exercise the HPE/Cray programming environment and evaluate the functionality and performance of the system. Acceptance testing of HPC11 required parallel execution of each element on Fawbush and Miller. In addition, careful coordination was needed to ensure successful acceptance of the newly deployed lustre file systems alongside the compute resources. In this work, we present test results from specific system components and provide an overview of the issues identified, challenges encountered, and the lessons learned along the way.

Melesse Vergara, Verónica G.↗

Overview of the Aeroelastic Prediction Workshop

The Aeroelastic Prediction Workshop brought together an international community of computational fluid dynamicists as a step in defining the state of the art in computational aeroelasticity. This workshop's technical focus was prediction of unsteady pressure distributions resulting from forced motion, benchmarking the results first using unforced system data. The most challenging aspects of the physics were identified as capturing oscillatory shock behavior, dynamic shock-induced separated flow and tunnel wall boundary layer influences. The majority of the participants used unsteady Reynolds-averaged Navier Stokes codes. These codes were exercised at transonic Mach numbers for three configurations and comparisons were made with existing experimental data. Substantial variations were observed among the computational solutions as well as differences relative to the experimental data. Contributing issues to these differences include wall effects and wall modeling, non-standardized convergence criteria, inclusion of static aeroelastic deflection, methodology for oscillatory solutions, post-processing methods. Contributing issues pertaining principally to the experimental data sets include the position of the model relative to the tunnel wall, splitter plate size, wind tunnel expansion slot configuration, spacing and location of pressure instrumentation, and data processing methods.

Heeg, Jennifer↗

A Benchmark Suite for Evaluating Scientific AI Workloads on GPUs

AI applications have been steadily increasing in the allocation portfolio among leadership computing facilities. These applications depend on deep learning frameworks with hardware acceleration and underlying software systems. With the rapid development of applications, software stacks, and hardware devices, it is essential to evaluate the performance of core operations in AI workloads for direction of optimizations and procurement of next-generation high-performance computing (HPC) infrastructures. Currently, most benchmarks lack scientific AI workloads. So, we present DeepKernelBench and the experimental results of evaluating the benchmark suite for early observations and performance comparisons on datacenter GPUs using representative workloads for scientific AI, including Attentions, General matrix multiplications, Geometrics and Fourier neural operations.

Jin, Zheming [Advanced Micro Devices (AMD)]↗

Performance modeling of microsecond scale biological molecular dynamics simulations on heterogeneous architectures

Performance improvements in biomolecular simulations based on molecular dynamics (MD) codes are widely desired. Unfortunately, the factors, which allowed past performance improvements, particularly the microprocessor clock frequencies, are no longer increasing. Hence, novel software and hardware solutions are being explored for accelerating performance of widely used MD codes. In this paper, we describe our efforts on porting, optimizing and tuning of Large-scale Atomic/Molecular Massively Parallel Simulator, a popular MD framework, on heterogeneous architectures: multi-core processors with graphical processing unit (GPU) accelerators. Our implementation is based on accelerating the most computationally expensive non-bonded interaction terms on the GPUs and overlapping the computation on the CPU and GPUs. This functionality is built on top of message passing interface that allows multi-level parallelism to be extracted even at the workstation level with the multi-core CPUs and allows extension of the implementation on GPU-enabled clusters. We hypothesize that the optimal benefit of heterogeneous architectures for applications will come by utilizing all possible resources (for example, CPU-cores and GPU devices on GPU-enabled clusters). Benchmarks for a range of biomolecular system sizes are provided, and an analysis is performed on four generations of NVIDIA's GPU devices. On GPU-enabled Linux clusters, by overlapping and pipelining computation and communication, we observe up to 10-folds application acceleration in multi-core and multi-GPU environments illustrating significant performance improvements. Detailed analysis of the implementation is presented that allows identification of bottlenecks in algorithm, indicating that code optimization and improvements on GPUs could allow microsecond scale simulation throughput on workstations and inexpensive GPU clusters, putting widely desired biologically relevant simulation time-scales within reach of a large user community. In order to systematically optimize simulation throughput and to enable performance prediction, we have developed a parameterized performance model that will allow developers and users to explore the performance potential of future heterogeneous systems for biological simulations.

Agarwal, Pratul K.↗

A Step Towards the Final Frontier: Lessons Learned from Acceptance Testing of the First HPE/Cray EX 3000 System at ORNL

In this paper, we summarize the deployment of the Air Force Weather (AFW) HPC11 system at Oak Ridge National Laboratory (ORNL) including the process followed to successfully complete acceptance testing of the system. HPC11 is the first HPE/Cray EX 3000 system that has been successfully released to its user community in a federal facility. HPC11 consists of two identical 800-node supercomputers, Fawbush and Miller, with access to two independent and identical Lustre parallel file systems. HPC11 is equipped with Slingshot 10 interconnect technology and relies on the HPE Performance Cluster Manager (HPCM) software for system configuration. ORNL has a clearly defined acceptance testing process used to ensure that every new system deployed can provide the necessary capabilities to support user workloads. We worked closely with HPE and AFW to develop a set of tests that used the United Kingdom’s Meteorological Office’s Unified Model (UM) and 4DVAR. We also included benchmarks and applications from the Oak Ridge Leadership Computing Facility (OLCF) portfolio to fully exercise the HPE/Cray programming environment and evaluate the functionality and performance of the system. Acceptance testing of HPC11 required parallel execution of each element on Fawbush and Miller. In addition, careful coordination was needed to ensure successful acceptance of the newly deployed Lustre file systems alongside the compute resources. In this work, we present test results from specific system components and provide an overview of the issues identified, challenges encountered, and the lessons learned along the way.

Melesse Vergara, Veronica↗

Appointing silver and bronze standards for noncovalent interactions: A comparison of spin-component-scaled (SCS), explicitly correlated (F12), and specialized wavefunction approaches

A systematic examination of noncovalent interactions as modeled by wavefunction theory is presented in comparison to gold-standard quality benchmarks available for 345 interaction energies of 49 bimolecular complexes. Quantum chemical techniques examined include spin-component-scaling (SCS) variations on second-order perturbation theory (MP2) [SCS, SCS(N), SCS(MI)] and coupled cluster singles and doubles (CCSD) [SCS, SCS(MI)]; also, method combinations designed to improve dispersion contacts [DW-MP2, MP2C, MP2.5, DW-CCSD(T)-F12]; where available, explicitly correlated (F12) counterparts are also considered. Dunning basis sets augmented by diffuse functions are employed for all accessible ζ-levels; truncations of the diffuse space are also considered. After examination of both accuracy and performance for 394 model chemistries, SCS(MI)-MP2/cc-pVQZ can be recommended for general use, having good accuracy at low cost and no ill-effects such as imbalance between hydrogen-bonding and dispersion-dominated systems or non-parallelity across dissociation curves. Moreover, when benchmarking accuracy is desirable but gold-standard computations are unaffordable, this work recommends silver-standard [DW-CCSD(T**)-F12/aug-cc-pVDZ] and bronze-standard [MP2C-F12/aug-cc-pVDZ] model chemistries, which support accuracies of 0.05 and 0.16 kcal/mol and efficiencies of 97.3 and 5.5 h for adenine·thymine, respectively. Choice comparisons of wavefunction results with the best symmetry-adapted perturbation theory [T. M. Parker, L. A. Burns, R. M. Parrish, A. G. Ryno, and C. D. Sherrill, J. Chem. Phys. 140, 094106 (2014)] and density functional theory [L. A. Burns, Á. Vázquez-Mayagoitia, B. G. Sumpter, and C. D. Sherrill, J. Chem. Phys. 134, 084107 (2011)] methods previously studied for these databases are provided for readers' guidance.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Blade Displacement Measurements of the Full-Scale UH-60A Airloads Rotor

Blade displacement measurements were acquired during a wind tunnel test of the full-scale UH-60A Airloads rotor. The test was conducted in the 40- by 80-Foot Wind Tunnel of the National Full-Scale Aerodynamics Complex at NASA Ames Research Center. Multi-camera photogrammetry was used to measure the blade displacements of the four-bladed rotor. These measurements encompass a range of test conditions that include advance ratios from 0.15 to unique slowed-rotor simulations as high as 1.0, thrust coefficient to rotor solidity ratios from 0.01 to 0.13, and rotor shaft angles from -10.0 to 8.0 degrees. The objective of these measurements is to provide a benchmark blade displacement database to be utilized in the development and validation of rotorcraft computational tools. The methodology, system development, measurement techniques, and preliminary sample blade displacement measurements are presented.

Barrows, Danny A.↗

Parallel Performance Characterization of Columbia

Using a collection of benchmark problems of increasing levels of realism and computational effort, we will characterize the strengths and limitations of the 10,240 processor Columbia system to deliver supercomputing value to application scientists. Scientists need to be able to determine if and how they can utilize Columbia to carry extreme workloads, either in terms of ultra-large applications that cannot be run otherwise (capability), or in terms of very large ensembles of medium-scale applications to populate response matrices (capacity). We select existing application benchmarks that scale from a small number of processors to the entire machine, and that highlight different issues in running supercomputing-calss applicaions, such as the various types of memory access, file I/O, inter- and intra-node communications and parallelization paradigms. http://www.nas.nasa.gov/Software/NPB/

Biswas, Rupak↗

Status of the CERBERUS Evaluation for the International Criticality Safety Benchmark Evaluation Project (ICSBEP) Handbook

Modeling & Simulation (M&S) tools are used to analyze advanced reactor designs and the safety of current nuclear operations. As computers continue to improve, we are able to enhance resolution in our calculations. Therefore, the limitations of simulation capability are in the quality of data that is being used, including our ability to quantify the uncertainty and sensitivity of that data. In order to model systems of interest with increasing accuracy, the industry must improve key nuclear data measurements. The International Criticality Safety Benchmark Evaluation Project (ICSBEP) compiles and evaluates experiment data in a handbook that can be used by criticality safety engineers and others to validate computer codes and cross section libraries at nuclear facilities. Both critical and subcritical experiments are included in the handbook. These experiments, along with differential measurements, can help improve the quality of nuclear data. Concerns regarding the accuracy of Cu nuclear data have been published. The large values and trend of C-E for the Zeus intermediate energy benchmark, being one of the primary examples. Furthermore, very few experiments have been designed to be sensitive to Cu (as shown in Figure 1), so an integral, critical experiment is needed to help resolve these differences.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Combining static analysis and state transition graphs for verification of event-condition-action systems in the RERS 2012 and 2013 challenges

In this work, we present a combination of approaches for the verification of event-condition-action (ECA) systems. The analyzed ECA systems range from structurally simple to structurally complex systems. We address the verification of reachability properties and behavioral properties. Reachability properties are represented by assertions in the program and we determine statically whether an assertion holds for all execution paths. Behavioral properties are represented as linear temporal logic formulas specifying the input/output behavior of the program. Our approach assumes a finite state space. We compare a symbolic analysis with an exhaustive state space exploration and discuss the trade-offs between the approaches in terms of the number of computed states and run-time behavior. All variants compute a state transition graph which can also be passed to an LTL verifier. The variants have a different impact on the number of computed states in the state transition graph which in turn impacts the run-time and memory consumption of subsequent phases. Finally, we evaluate the different analysis variants with the RERS benchmarks.

97 MATHEMATICS AND COMPUTING↗

Benchmark Specification for FFTF LOFWOS Test #13

The Fast Flux Test Facility (FFTF) at the Hanford site in Washington was designed by the Westinghouse Electric Corporation for the U.S. Department of Energy. FFTF was a 400 MW thermal, oxide-fueled, liquid sodium cooled test reactor, built to assist development and testing of advanced fuels and materials for fast breeder reactors. After reaching criticality in 1980, FFTF operated until 1992, providing the U.S. Department of Energy (DOE) with the means to test fuels, materials, and other components in a fast neutron flux environment. In July 1986, a series of unprotected transients (with the plant protection system intentionally disabled) were performed in FFTF as part of the passive safety demonstration program. Among these were thirteen loss of flow without scram (LOFWOS) tests. The goals of this program included confirming the liquid metal reactor safety margins, providing data for computer code validation, and demonstrating the inherent and passive safety benefits of specific design features. The test defined in this benchmark is LOFWOS Test #13, which was initiated at 50% power and 100% flow with the pump pony motors turned off. This benchmark specification is intended to support collaborative efforts within international partnerships on the validation of simulation tools and models in the area of Sodium-cooled Fast Reactor (SFR) safety. Validated tools and models are needed to evaluate SFR inherent safety characteristics and assess the impact of passive design features in response to accident initiators. Comparisons with experimental data and the results of safety analyses from other groups create unique opportunities to improve predictive capabilities of computational codes and methods for SFR modeling and simulation. The conditions of the LOFWOS test along with the feedback from FFTF’s limited free bow core restraint system and the novel passive reactivity control Gas-Expansion Modules (GEMs) pose a very challenging and uniquely valuable benchmark exercise.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Optimization Algorithms as Quantum Performance Benchmarks

Combinatorial optimization is anticipated to be one of the primary use cases for quantum computation in the coming years. The Quantum Approximate Optimization Algorithm (QAOA) and Quantum Annealing (QA) have the potential to demonstrate significant run-time performance benefits over current state-of-the-art solutions. Using existing methods for characterizing classical optimization algorithms, we analyze solution quality obtained by solving Max-Cut problems using a quantum annealing device and gate-model quantum simulators and devices. This is used to guide the development of an advanced benchmarking framework for quantum computers designed to evaluate the trade-off between run-time execution performance and the solution quality for iterative hybrid quantum-classical applications. The framework generates performance profiles through effective visualizations that show performance progression as a function of time for various problem sizes and illustrates algorithm limitations uncovered by the benchmarking approach. The framework is an enhancement to the existing open-source QED-C Application-Oriented Benchmark suite and can connect to the open-source analysis libraries. The suite can be executed on various quantum simulators and quantum hardware systems.

benchmarking↗