Search NASA⌕ Search

SEARCH · Search NASA

Results for “Performance Benchmarking”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,531 records · Page 85

Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm

Abstract We present a benchmark test suite and an automated machine learning procedure for evaluating supervised machine learning (ML) models for predicting properties of inorganic bulk materials. The test suite, Matbench, is a set of 13 ML tasks that range in size from 312 to 132k samples and contain data from 10 density functional theory-derived and experimental sources. Tasks include predicting optical, thermal, electronic, thermodynamic, tensile, and elastic properties given a material’s composition and/or crystal structure. The reference algorithm, Automatminer, is a highly-extensible, fully automated ML pipeline for predicting materials properties from materials primitives (such as composition and crystal structure) without user intervention or hyperparameter tuning. We test Automatminer on the Matbench test suite and compare its predictive power with state-of-the-art crystal graph neural networks and a traditional descriptor-based Random Forest model. We find Automatminer achieves the best performance on 8 of 13 tasks in the benchmark. We also show our test suite is capable of exposing predictive advantages of each algorithm—namely, that crystal graph methods appear to outperform traditional machine learning methods given ~10 4 or greater data points. We encourage evaluating materials ML algorithms on the Matbench benchmark and comparing them against the latest version of Automatminer.

36 MATERIALS SCIENCE↗

Verification of MCNP5-1.60 and MCNP6-Beta-2 for Criticality Safety Applications

To verify that both MCNP5-1.60 and MCNP6-Beta-2 are performing correctly for criticality safety applications, several suites of verification/validation benchmark problems were run in early 2012. Results from these benchmark suites were compared with results from previously verified versions of MCNP5. The goals of this verification testing were: (1) Verify that MCNP5-1.60 works correctly for nuclear criticality safety applications, producing the same results as for the previous verification performed in 2010; (2) Determine the sensitivity to computer roundoff using different Fortran-90 compilers for building MCNP5 and MCNP6, to support moving to current versions of the compilers; and (3) Verify that MCNP6-Beta-2 works correctly for nuclear criticality safety applications, producing the same results as for MCNP5-1.60. This provides support for eventual migration of users and applications to MCNP6. The current production version of MCNP5 included in the RSICC release package is MCNP5-1.60. This version was first distributed by RSICC in October 2010. While there were subsequent RSICC distributions of the MCNP package in July 2011 and February 2012, no changes were made to MCNP5-1.60. The RSICC release package in February 2012 included both MCNP5-1.60 and the current beta version of MCNP6, MCNP6-Beta-2. MCNP6 is the merger of MCNP5 and MCNPX capabilities. The current release of MCNP6 available from RSICC as of February 2012 is MCNP6-Beta-2. This version includes all of the features for criticality safety calculations that are available in MCNP5-1.60, and many new features largely unrelated to nuclear criticality safety calculations. This release is a 'beta' release to allow intermediate and advanced users to begin testing the merged code in their field of expertise. It should not be used for production calculations.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Milestone M1 Report: HBM2/3 Evaluation on Many-core CPU WBS 2.4, Milestone ECP-MT-1000

In HIHE01-1, "Evaluate a PathForward/Facilities memory-relevant performance study/analysis," we conducted a focused study on the performance differences between HBM2 and HBM3 as revealed through execution of representative benchmarks. We used measurements on an existing many-core system, Knight's Landing (KNL), to calibrate simulator settings, and then performed Structural Simulation Toolkit (SST) simulations of KNL-like CPUs that access future high bandwidth memories. This report documents our findings.

97 MATHEMATICS AND COMPUTING↗

SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructions

Instruction finetuning is a popular paradigm to align large language models (LLM) with human intent. Despite its popularity, this idea is less explored in improving the LLMs to align existing foundation models with scientific disciplines, concepts and goals. In this work, we present SciTune as a tuning framework to improve the ability of LLMs to follow scientific multimodal instructions. To test our methodology, we use a human-generated scientific instruction tuning dataset and train a large multimodal model LLaMA-SciTune that connects a vision encoder and LLM for science-focused visual and language understanding. LLaMA-SciTune significantly outperforms the state-of-the-art models in the generated figure types and captions in multiple scientific multimodal benchmarks. In comparison to the models that are fine-tuned with machine generated data only, LLaMA-SciTune surpasses human performance on average and in many sub-categories on the ScienceQA benchmark.

• Artificial intelligence (AI) / machine learning ↗

Benchmarking Autonomous Scattering Experiments Illustrated on TAS

With the advancement of artificial intelligence and machine learning methods, autonomous approaches are recognized to have great potential for performing more efficient scattering experiments. In our view, it is crucial for such approaches to provide thorough evidence about respective performance improvements in order to increase acceptance within a scientific community. Therefore, we propose a benchmarking procedure designed as a cost-benefit analysis that is applicable to any scattering method sequentially collecting data during an experiment. For a given approach, the performance assessment is based on how much benefit, given a certain cost budget, it is able to acquire in predefined test cases. Different approaches thus get a chance for comparison and can make their advantages explicit and visible. Key components of the procedure, i.e., cost measures, benefit measures, and test cases, are made precise for the setting of three-axes spectrometry (TAS) as an illustration. Finally, we discuss neglected aspects and possible extensions for the TAS setting and comment on the procedure’s applicability to other scattering methods. A Python implementation of the procedure to simplify its utilization by interested researchers from the field is also provided.

36 MATERIALS SCIENCE↗

Low Molecular Weight Polymethacrylates as Multi-Functional Lubricant Additives

Low molecular weight, moderately polar polymethacrylate polymers are explored as potential multi-functional lubricant additives. The performance of these novel additives in base oil is characterized in terms of their viscosity index, shear stability, friction and wear, and compared to two benchmarks, a base oil containing a typical polymeric viscosity modifier and a fully-formulated oil. Results show that the best performing of these polymers exhibit viscosity index and friction comparable to that of both benchmarks, far superior shear stability to either benchmark (as much as 15x lower shear), and wear reduction significantly better than a typical viscosity modifier (lower by a factor of 2-3) but not as good as that of the fully formulated oil. The findings also suggest that the polarity and molecular weight of the polymers affect their performance which suggests future synthetic strategies which may enable this new class of additives to replace multiple additives in typical lubricant formulations.

Cosimbescu, Lelia↗

ConnectX2 In niBand Management Queues: New support for Network Of oaded

This paper introduces the newly developed InfiniBand (IB) Management Queue capability, used by the Host Channel Adapter (HCA) to manage network task data flow dependancies, and progress the communications associated with such flows. These tasks include sends, receives, and the newly supported wait task, and are scheduled by the HCA based on a data dependency description provided by the user. This functionality is supported by the ConnectX-2 HCA, and provides the means for delegating collective communication management and progress to the HCA, also known as collective communication offload. This provides a means for overlapping collective communications managed by the HCA and computation on the Central Processing Unit (CPU), thus making it possible to reduce the impact of system noise on parallel applications using collective operations. This paper further describes how this new capability can be used to implement scalable Message Passing Interface (MPI) collective operations, describing the high level details of how this new capability is used to implement the MPI Barrier collective operation, focusing on the latency sensitive performance aspects of this new capability. This paper concludes with small scale benchmark experiments comparing implementations of the barrier collective operation, using the new network offload capabilities, with established point-to-point based implementations of these same algorithms, which manage the data flow using the central processing unit. These early results demonstrate the promise this new capability provides to improve the scalability of high-performance applications using collective communications. The latency of the HCA based implementation of the barrier is similar to that of the best performing point-to-point based implementation managed by the central processing unit, starting to outperform these as the number of processes involved in the collective operation increases.

99 GENERAL AND MISCELLANEOUS↗

Assessment of four strain energy decomposition methods for phase field fracture models using quasi-static and dynamic benchmark cases

Abstract Strain energy decomposition methods in phase field fracture models separate strain energy that contributes to fracture from that which does not. However, various decomposition methods have been proposed in the literature, and it can be difficult to determine an appropriate method for a given problem. The goal of this work is to facilitate the choice of strain decomposition method by assessing the performance of three existing methods (spectral decomposition of the stress or the strain and deviatoric decomposition of the strain) and one new method (deviatoric decomposition of the stress) with several benchmark problems. In each benchmark problem, we compare the performance of the four methods using both qualitative and quantitative metrics. In the first benchmark, we compare the predicted mechanical behavior of cracked material. We then use four quasi-static benchmark cases: a single edge notched tension test, a single edge notched shear test, a three-point bending test, and a L-shaped panel test. Finally, we use two dynamic benchmark cases: a dynamic tensile fracture test and a dynamic shear fracture test. All four methods perform well in tension, the two spectral methods perform better in compression and with mixed mode (though the stress spectral method performs the best), and all the methods show minor issues in at least one of the shear cases. In general, whether the strain or the stress is decomposed does not have a significant impact on the predicted behavior.

Zhang, Shuaifang↗

Intercomparison of flood inundation models across land use types and hydrological flood stages

Flood Inundation Mapping (FIM) model selection is a key operational decision because accurate, rapid mapping underpins early warning and resource allocation. FIM performance is context-dependent and can vary with hydrograph phase, land-use/land-cover (LULC), and the evaluation benchmark. Intercomparison studies typically assess a single near-peak snapshot against one reference dataset. Here, we provide a context-stratified intercomparison across (i) multiple hydrograph phases, (ii) LULC classes, and (iii) benchmark types, for five FIM approaches spanning a wide range of physical complexity and operational cost (TRITON, LISFLOOD-FP, HEC-RAS 2D, ARC-Curve2Flood, and OWP HAND-FIM). We use the Hurricane Matthew flood (2016) in the Neuse River Basin, North Carolina, USA, as a case study. Using high-resolution remote sensing-derived flood inundation maps, hand-labeled points, and building footprints, we assess model skill across two rising and two falling hydrograph limbs and across major LULC types. Results show that model rankings shift systematically across contexts: LISFLOOD-FP ranks highest in three of four flood phases, while TRITON leads during one rising limb phase; LISFLOOD-FP performs best in vegetated areas, whereas HEC-RAS improves relative performance in agricultural and urban areas; and benchmark choice influences conclusions, with LISFLOOD-FP performing best for flooded-building detection in the late falling limb, while TRITON ranks highest against hand-labeled points. We also report representative wall-clock runtimes for each workflow to provide use-case context for operational feasibility. Together, these results offer transferable guidance for model selection and for designing large-scale, benchmark-aware FIM intercomparison studies.

Nikrou, Parvaneh [University of Alabama]↗

Performance, Efficiency, and Effectiveness of Supercomputers

Our primary purpose here is to offer to the general technical and policy audience a perspective on whether the supercomputing community should focus on improving the efficiency of supercomputing systems and their use rather than on building larger and ostensibly more capable systems that are used at low efficiency. After first summarizing our content and defining some necessary terms, we give a concise answer to this question. We then set this in context by characterizing performance of current supercomputing systems on a variety of benchmark problems and actual problems drawn from workloads in the national security, industrial, and scientific context. We also answer some related questions, identify some important technological trends, and offer a perspective on the significance of these trends. We hope by doing so to better equip the reader to evaluate commentary and controversy concerning supercomputing performance.

97 MATHEMATICS AND COMPUTING↗

Summary of ORSphere Critical and Reactor Physics Measurements

In the early 1970s Dr. John T. Mihalczo (team leader), J. J. Lynn, and J. R. Taylor performed experiments at the Oak Ridge Critical Experiments Facility (ORCEF) with highly enriched uranium (HEU) metal (called Oak Ridge Alloy or ORALLOY) to recreate GODIVA I results with greater accuracy than those performed at Los Alamos National Laboratory in the 1950s. The purpose of the Oak Ridge ORALLOY Sphere (ORSphere) experiments was to estimate the unreflected and unmoderated critical mass of an idealized sphere of uranium metal corrected to a density, purity, and enrichment such that it could be compared with the GODIVA I experiments. This critical configuration has been evaluated. Preliminary results were presented at ND2013. Since then, the evaluation was finalized and judged to be an acceptable benchmark experiment for the International Criticality Safety Benchmark Experiment Project (ICSBEP). Additionally, reactor physics measurements were performed to determine surface button worths, central void worth, delayed neutron fraction, prompt neutron decay constant, fission density and neutron importance. These measurements have been evaluated and found to be acceptable experiments and are discussed in full detail in the International Handbook of Evaluated Reactor Physics Benchmark Experiments. The purpose of this paper is summary summarize all the critical and reactor physics measurements evaluations and, when possible, to compare them to GODIVA experiment results.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Nuclear Data and Cross Section Testing Using ENDF/B-VIII.0

With the release of the Evaluated Nuclear Data File (ENDF)/B-VIII.0 library, nuclear criticality safety practitioners and engineers have access to the latest cross section sets available for their analyses. However, these cross sections must be rigorously tested and validated to ensure that the nuclear data are responsive to the needs of the individuals responsible for developing, implementing, and maintaining computational tools for criticality safety applications. Thus, the ENDF/B-VIII.0 library is tested and validated with a large collection of experiments that were vetted by the International Criticality Safety Benchmark Evaluation Project and made available in the International Handbook of Evaluated Criticality Safety Benchmark Experiments. A selection of benchmark experiments for use within the criticality safety community were prepared and reviewed within the Verified, Archived Library of Inputs and Data (VALID), which is maintained by the Nuclear Energy and Fuel Cycle Division at Oak Ridge National Laboratory. The performance of the ENDF/B-VIII.0 library is assessed by using VALID models of benchmark experiments with the beta 12 version of SCALE 6.3 KENO V.a and KENO-VI Monte Carlo codes. The performance is compared with the results obtained from with the ENDF/B-VII.1 library. This report considers multigroup (MG) and continuous energy (CE) formats of the ENDF/B-VIII.0 and -VII.1 libraries. The benchmark experiments within VALID that validate the ENDF/B-VIII.0 library cover 15 broad system categories by using a range of fissile materials, uranium enrichments, plutonium isotopic vectors, and mixed uranium/plutonium systems. These forms are represented as metals, solutions, or various arrays of rods or plates that cover a variety of neutron energy spectra: thermal, fast, mixed, and intermediate. Over 600 cases were considered for use with the KENO V.a and KENO-VI codes with the ENDF/B-VIII.0 library. The results of the Monte Carlo comparison of ENDF/B-VIII.0 to ENDF/B-VII.1 with both KENO V.a and KENO-VI indicate that there is a less than 0.53% Δk difference between the bias of calculated k eff from the expected values. The CE ENDF/B-VIII.0 library results in smaller magnitude biases than the ENDF/B-VII.1 data for HEU-MET-FAST, HEU-SOL-THERM, IEU-MET-FAST, LEU-SOL-THERM, PU-SOL-THERM, and U233-MET-FAST systems, while the MG results yielded smaller magnitude biases for HEU-MET-FAST, HEU-SOL-THERM, IEU-MET-FAST, LEU-COMP-THERM, LEU-SOL THERM, MIX-COMP-FAST, and U233-MET-FAST systems. Most notable are the adjustments to the plutonium and 233 U cross section data, which has resulted in noticeably lower biases in the ENDF/B VIII.0 results for the mixed, plutonium, and 233 U systems. Results of the sensitivity data file comparison generated from TSUNAMI-3D for selected VALID cases for the ENDF/B-VIII.0 library indicate a very high level of agreement with correlation coefficients of the effect of nuclear data uncertainty on k eff (the c k integral parameter) all above 0.99. This indicates that cases with the ENDF/B-VIII.0 library would see very similar responses to any nuclear data errors or change as those with the ENDF/B-VII.1 library.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Benchmarks for Model Validation based on LiDAR Wake Measurements

Technology development and design decisions in wind energy are often based on results from simulations performed for individual wind turbines or entire wind plants. It is therefore critical to ensure that the models being used for research and industry applications in wind energy be thoroughly validated against measurements. A full-system validation of wind plant simulations must consider the atmospheric inflow, the response of the wind turbines, and their wakes. This task is complicated by the lack of freely available, quality-controlled, high-quality measurements. Here, such measurements are used to offer a validation exercise that can be used to assess the accuracy of models of any fidelity level. When it comes to real-world measurements, the dataset considered herein is simple in terms of terrain but exhibits pronounced diurnal cycles. Instead of a full-scale wind plant, we consider an individual research-scale, utility wind turbine instrumented for power and loads measurements. Three benchmarks are defined, with increasing levels of complexity: near neutral, slightly unstable, and very stable atmospheric stratification. Through comparisons between observations and simulations, the benchmarks provide complementary information about the model performance and its ability to reproduce mean and dynamic wake characteristics. This article describes the measurements and methodology used to define these benchmarks and provides the information required to perform simulations and conduct the model-measurement comparison. The objective is to provide a robust wake model validation exercise open to anyone, which will serve to minimize uncertainty in model validation practices related to varying methodologies across simulation tools and users.

17 WIND ENERGY↗

Exploring the performance of spatial stochastic simulation algorithms

Since the publication of Gillespie's direct method, diverse methods have been developed to improve the performance of stochastic simulation methods and to enter the spatial realm. In this paper we discuss a spatial {tau}-leaping variant (S{tau}) that extends the basic leap method. S{tau} takes reaction and both outgoing and incoming diffusion events into account when calculating a leap candidate. A performance analysis shall reveal details on the achieved success in balancing speed and accuracy in comparison to other methods. However, performance analysis of spatial stochastic algorithms requires significant effort - it is crucial to choose suitable (benchmark) models and to carefully define model and simulation setups that take problem and simulation design spaces into account.

97 MATHEMATICS AND COMPUTING↗

CLC: A cross-level program characterization method

Characterization of program execution plays a key role in performance improvement. There are numerous transformations applied to each step that a program takes on its lowering from source code to a compiler intermediate representation to machine language to microarchitecture-specific execution. The unpredictable benefit of each transformation step could lead a notionally superior algorithm to exhibit inferior performance once actually run, and it can be hard to discern which step in the transformation path contradicted the code developer’s assumptions. Conventional approaches to program-execution characterization consider the behavior after only a single one of those steps, which limits the information that can be provided to the user. To help address the issue of myopic views of program execution, this paper presents a novel cross-level characterization approach for understanding the behavior of program execution at different levels in the process of writing, compiling, and running a program. We show that this approach provides a richer view of the sources of performance gains and losses and helps identify program execution in a more accurate manner.

97 MATHEMATICS AND COMPUTING↗

Impact of Microcrack Generation and Surface Degradation on a Nickel-Rich Layered Li[Ni 0.9 Co 0.05 Mn 0.05 ]O 2 Cathode for Lithium-Ion Batteries

In this work, to address the growing demand for energy density, the Ni-rich layered [Ni 0.90 Co 0.05 Mn 0.05 ]O 2 cathode has been synthesized and its electrochemical performance in lithium-ion cells has been benchmarked against a lower-Ni content Li[Ni 0.6 Co 0.2 Mn 0.2 ]O 2 cathode. Li[Ni 0.90 Co 0.05 Mn 0.05 ]O 2 delivers a high discharge capacity of 227 mA h g -1 compared to a capacity of 189 mA h g -1 for Li[Ni 0.6 Co 0.2 Mn 0.2 ]O 2 when cycled up to a lower cutoff voltage of 4.3 V, making it an appealing candidate for electric vehicles. With an increase in the charge cutoff voltage to 4.5 V, Li[Ni 0.90 Co 0.05 Mn 0.05 ]O 2 displays a capacity of 238 mA h g -1 compared to a capacity of 208 mA h g -1 for Li[Ni 0.6 Co 0.2 Mn 0.2 ]O 2 . Although Li[Ni 0.90 Co 0.05 Mn 0.05 ]O 2 suffers during cycling from the usual rapid capacity fade in a manner similar to that of LiNiO 2 , 87 and 81% of the initial capacity could still be retained after 100 cycles even after cycling to higher cutoff voltages of 4.3 and 4.5 V, respectively. A comparison of Li[Ni 0.90 Co 0.05 Mn 0.05 ]O 2 and Li[Ni 0.6 Co 0.2 Mn 0.2 ]O 2 reveals that the capacity fade of Li[Ni 0.90 Co 0.05 Mn 0.05 ]O 2 originates largely from the anisotropic volume change and subsequent microcrack propagation in the bulk and NiO-like rock salt impurity phase formation on the particle surface, which are exacerbated at 4.5 V. Lastly, future work with appropriate doping and surface modification could improve further the performance of Li[Ni 0.90 Co 0.05 Mn 0.05 ]O 2 .

25 ENERGY STORAGE↗

SU-F-T-152: Experimental Validation and Calculation Benchmark for a Commercial Monte Carlo Pencil BeamScanning Proton Therapy Treatment Planning System in Heterogeneous Media

Purpose: Eclipse AcurosPT 13.7, the first commercial Monte Carlo pencil beam scanning (PBS) proton therapy treatment planning system (TPS), was experimentally validated for an IBA dedicated PBS nozzle in the CIRS 002LFC thoracic phantom. Methods: A two-stage procedure involving the use of TOPAS 1.3 simulations was performed. First, Geant4-based TOPAS simulations in this phantom were experimentally validated for single and multi-spot profiles at several depths for 100, 115, 150, 180, 210 and 225 MeV proton beams, using the combination of a Lynx scintillation detector and a MatriXXPT ionization chamber array. Second, benchmark calculations were performed with both AcurosPT and TOPAS in a phantom identical to the CIRS 002LFC, with the exception that the CIRS bone/mediastinum/lung tissues were replaced with similar tissues that are predefined in AcurosPT (a limitation of this system which necessitates the two stage procedure). Results: Spot sigmas measured in tissue were in agreement within 0.2 mm of TOPAS simulation for all six energies, while AcurosPT was consistently found to have larger spot sigma (<0.7 mm) than TOPAS. Using absolute dose calibration by MatriXXPT, the agreements between profiles measurements and TOPAS simulation, and calculation benchmarks are over 97% except near the end of range using 2 mm/2% gamma criteria. Overdosing and underdosing were observed at the low and high density side of tissue interfaces, respectively, and these increased with increasing depth and decreasing energy. Near the mediastinum/lung interface, the magnitude can exceed 5 mm/10%. Furthermore, we observed >5% quenching effect in the conversion of Lynx measurements to dose. Conclusion: We recommend the use of an ionization chamber array in combination with the scintillation detector to measure absolute dose and relative PBS spot characteristics. We also recommend the use of an independent Monte Carlo calculation benchmark for the commissioning of a commercial TPS. Partially supported by Varian Medical System under the master agreement between Varian and University of pennsylvania.

60 APPLIED LIFE SCIENCES↗

Scalable Metropolis Monte Carlo for simulation of hard shapes

We design and implement a scalable hard particle Monte Carlo simulation toolkit (HPMC), and release it open source as part of HOOMD-blue. HPMC runs in parallel on many CPUs and many GPUs using domain decomposition. We employ BVH trees instead of cell lists on the CPU for fast performance, especially with large particle size disparity, and optimize inner loops with SIMD vector intrinsics on the CPU. Our GPU kernel proposes many trial moves in parallel on a checkerboard and uses a block-level queue to redistribute work among threads and avoid divergence. HPMC supports a wide variety of shape classes, including spheres/disks, unions of spheres, convex polygons, convex spheropolygons, concave polygons, ellipsoids/ellipses, convex polyhedra, convex spheropolyhedra, spheres cut by planes, and concave polyhedra. NVT and NPT ensembles can be run in 2D or 3D triclinic boxes. Additional integration schemes permit Frenkel–Ladd free energy computations and implicit depletant simulations. In a benchmark system of a fluid of 4096 pentagons, HPMC performs 10 million sweeps in 10 min on 96 CPU cores on XSEDE Comet. The same simulation would take 7.6 h in serial. HPMC also scales to large system sizes, and the same benchmark with 16.8 million particles runs in 1.4 h on 2048 GPUs on OLCF Titan.

Computer Science↗