Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmarking Software”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

The experiment filtering tool in the MACSENS workbench - 14341

The MACSENS (outils Moret d'Aide a la Caracterisation et aux etudes de SENSibilite) workbench is dedicated to the advanced validation work for criticality studies. The aim of this kind of tool is to determine the experiments from a validation database being the 'closest' to an application case and, from there, to evaluate a calculation bias with a related uncertainty. To achieve this goal, the MACSENS workbench relies on different kind of information stored in various databases depending on the origin of the data. Two main kinds of data can be identified. The first one concerns the data used to describe the experiments of the validation database (e.g. the inputs needed to perform a calculation). These values characterize the chemical and geometrical models. They can be considered as constant, as they do not vary once evaluated, and some of them can be from a non-numerical nature (reflector material for example...). The second kind of data gathers the results obtained with the use of a simulation code, such as reaction rates, fluxes and leakages, sensitivities, k{sub eff}, etc. These values depend on the version of the code and are purely numerical. In the early 2000's, regarding the increasing number of experiments evaluated in the ICSBEP handbook, the OECD/NEA (Organization for Economic Co-operation and Development/Nuclear Energy Agency), under the ICSBEP (International Criticality Safety Benchmark Evaluation Project) working group direction, decided to store such information in a database and to provide a software, called DICE (Database for the International Criticality safety benchmark Evaluation project), allowing users to search easily criticality experiments of interest. Up to now, more than 4400 critical or subcritical configurations have been compiled by ICSBEP, described and stored in the DICE database. Over time, a validation work has been done on the data, with the participation of IRSN, and a quality assurance policy has been defined. For these reasons, the DICE database is highly reliable. These features were fundamental for the MACSENS developers, who were looking for a database containing descriptive data of the experiments included in the MORET code validation database. The paper describes the solution chosen by IRSN to take advantage of the DICE database content and to overcome the DICE limits. It also explains the quality assurance policy applied to guaranty the reliability of the stored data. It will also discuss the use of the experiment-filtering module of the MACSENS workbench to select a set of experiments and to examine numerical parameter trends for an application case. (authors)

42 ENGINEERING↗

Benchmarks for GADRAS performance validation.

The performance of the Gamma Detector Response and Analysis Software (GADRAS) was validated by comparing GADRAS model results to experimental measurements for a series of benchmark sources. Sources for the benchmark include a plutonium metal sphere, bare and shielded in polyethylene, plutonium oxide in cans, a highly enriched uranium sphere, bare and shielded in polyethylene, a depleted uranium shell and spheres, and a natural uranium sphere. The benchmark experimental data were previously acquired and consist of careful collection of background and calibration source spectra along with the source spectra. The calibration data were fit with GADRAS to determine response functions for the detector in each experiment. A one-dimensional model (pie chart) was constructed for each source based on the dimensions of the benchmark source. The GADRAS code made a forward calculation from each model to predict the radiation spectrum for the detector used in the benchmark experiment. The comparisons between the GADRAS calculation and the experimental measurements are excellent, validating that GADRAS can correctly predict the radiation spectra for these well-defined benchmark sources.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

OpenMxP-Opensource Mixed Precision Computing

This is an opensource library for benchmarking the system's GPU mixed precision capabilities. The software calculates solution of the system of linear equation in 64bit accuracy using mixed precision techniques and iterative refinement. Original benchmark designed is done by ICL, and it is name HPL-MxP (HPL-AI)

Lu, Hao↗

miniVite: A Graph Analytics Benchmarking Tool for Massively Parallel Systems

Benchmarking of high performance computing sys-tems can help provide critical insights for ef?cient design of computing systems and software applications. Although a large number of tools for benchmarking exist, there is a lack of rep-resentative benchmarks for the class of irregular computations as exempli?ed by graph analytics. In this paper, we propose miniVite as a representative graph analytics benchmark tool to test a variety of distributed-memory systems. Graph clustering, popularly known as community detection, is a prototypical graph operation used in numerous scienti?c computing and analytics applications. The goal of clustering is to partition a graph into clusters (or communities) such that each cluster consists of vertices that are densely connected within the cluster and sparsely connected to the rest of the graph. Modu-larity optimization is a popular technique for identifying clusters in a graph. Ef?cient parallelization of modularity optimization-based algorithms is challenging. One successful approach was conceived in Vite, a distributed-memory implementation of the Louvain algorithm that incorporates several heuristics. We introduce miniVite as a representative but simpli?ed variant of Vite, to serve as a prototypical graph analytics benchmarking tool. Unlike other graph-based methods such as breadth-?rst search and betweenness centrality, miniVite represents highly complex computational patterns stressing a variety of system features, which can provide crucial insight for co-design of future computing systems.

community detection, parallel, MPI↗

Structural Analysis Methodology for Space Deployable Structures Using Multi-Body Dynamic Solver

The purpose of this paper is to provide detailed structural analysis implementation for space deployable structures using a multi-body dynamic solver. The selected commercial code is MSC/ADAMS. Four benchmark problems are used to evaluate mechanism motion, frictional contacts, and geometrically non-linear large deformation. This paper examines the software functionality, performance, accuracy and ability to solve the benchmark problems. The discussions covered are related to solver convergence, tracking internal force, velocity and acceleration of different parts in each model and comparing results to closed form solutions if available.

Montazersadgh, Farzin↗

Experimental Evaluation of Verification and Validation Tools on Martian Rover Software

To achieve its science objectives in deep space exploration, NASA has a need for science platform vehicles to autonomously make control decisions in a time frame that excludes intervention from Earth-based controllers. Round-trip light-time is one significant factor motivating autonomy capability, another factor is the need to reduce ground support operations cost. An unsolved problem potentially impeding the adoption of autonomy capability is the verification and validation of such software systems, which exhibit far more behaviors (and hence distinct execution paths in the software) than is typical in current deepspace platforms. Hence the need for a study to benchmark advanced Verification and Validation (V&V) tools on representative autonomy software. The objective of the study was to access the maturity of different technologies, to provide data indicative of potential synergies between them, and to identify gaps in the technologies with respect to the challenge of autonomy V&V. The study consisted of two parts: first, a set of relatively independent case studies of different tools on the same autonomy code, second a carefully controlled experiment with human participants on a subset of these technologies. This paper describes the second part of the study. Overall, nearly four hundred hours of data on human use of three different advanced V&V tools were accumulated, with a control group that used conventional testing methods. The experiment simulated four independent V&V teams debugging three successive versions of an executive controller for a Martian Rover. Defects were carefully seeded into the three versions based on a profile of defects from CVS logs that occurred in the actual development of the executive controller. The rest of the document is structured a s follows. In section 2 and 3, we respectively describe the tools used in the study and the rover software that was analyzed. In section 4 the methodology for the experiment is described; this includes the code preparation, seeding of defects, participant training and experimental setup. Next we give a qualitative overview of how the experiment went from the point of view of each technology; model checking (section 5), static analysis (section 6), runtime analysis (section 7) and testing (section 8). The find section gives some preliminary quantitative results on how the tools compared.

Brat, Guillaume↗

Opportunities for enhancing MLCommons efforts while leveraging insights from educational MLCommons earthquake benchmarks efforts

MLCommons is an effort to develop and improve the artificial intelligence (AI) ecosystem through benchmarks, public data sets, and research. It consists of members from start-ups, leading companies, academics, and non-profits from around the world. The goal is to make machine learning better for everyone. In order to increase participation by others, educational institutions provide valuable opportunities for engagement. In this article, we identify numerous insights obtained from different viewpoints as part of efforts to utilize high-performance computing (HPC) big data systems in existing education while developing and conducting science benchmarks for earthquake prediction. As this activity was conducted across multiple educational efforts, we project if and how it is possible to make such efforts available on a wider scale. This includes the integration of sophisticated benchmarks into courses and research activities at universities, exposing the students and researchers to topics that are otherwise typically not sufficiently covered in current course curricula as we witnessed from our practical experience across multiple organizations. As such, we have outlined the many lessons we learned throughout these efforts, culminating in the need for benchmark carpentry for scientists using advanced computational resources. The article also presents the analysis of an earthquake prediction code benchmark while focusing on the accuracy of the results and not only on the runtime; notedly, this benchmark was created as a result of our lessons learned. Energy traces were produced throughout these benchmarks, which are vital to analyzing the power expenditure within HPC environments. Additionally, one of the insights is that in the short time of the project with limited student availability, the activity was only possible by utilizing a benchmark runtime pipeline while developing and using software to generate jobs from the permutation of hyperparameters automatically. It integrates a templated job management framework for executing tasks and experiments based on hyperparameters while leveraging hybrid compute resources available at different institutions. The software is part of a collection called cloudmesh with its newly developed components, cloudmesh-ee (experiment executor) and cloudmesh-cc (compute coordinator).

58 GEOSCIENCES↗

FastML Science Benchmarks: Accelerating Real-Time Scientific Edge Machine Learning

Applications of machine learning (ML) are growing by the day for many unique and challenging scientific applications. However, a crucial challenge facing these applications is their need for ultra low-latency and on-detector ML capabilities. Given the slowdown in Moore's law and Dennard scaling, coupled with the rapid advances in scientific instrumentation that is resulting in growing data rates, there is a need for ultra-fast ML at the extreme edge. Fast ML at the edge is essential for reducing and filtering scientific data in real-time to accelerate science experimentation and enable more profound insights. To accelerate real-time scientific edge ML hardware and software solutions, we need well-constrained benchmark tasks with enough specifications to be generically applicable and accessible. These benchmarks can guide the design of future edge ML hardware for scientific applications capable of meeting the nanosecond and microsecond level latency requirements. To this end, we present an initial set of scientific ML benchmarks, covering a variety of ML and embedded system techniques.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

WfBench: Automated Generation of Scientific Workflow Benchmarks

The prevalence of scientific workflows with high computational demands calls for their execution on various distributed computing platforms, including large-scale leadership-class high-performance computing (HPC) clusters. To handle the deployment, monitoring, and optimization of workflow executions, many workflow systems have been developed over the past decade. There is a need for workflow benchmarks that can be used to evaluate the performance of workflow systems on current and future software stacks and hardware platforms.We present a generator of realistic workflow benchmark specifications that can be translated into benchmark code to be executed with current workflow systems. Our approach generates workflow tasks with arbitrary performance characteristics (CPU, memory, and I/O usage) and with realistic task dependency structures based on those seen in production workflows. We present experimental results that show that our approach generates benchmarks that are representative of production workflows, and conduct a case study to demonstrate the use and usefulness of our generated benchmarks to evaluate the performance of workflow systems under different configuration scenarios.

Coleman, Taina↗

Distributed Software-Defined Network Architecture for Smart Grid Resilience to Denial-of-Service Attacks

An important challenge for smart grid security is designing a secure and robust smart grid communications architecture to protect against cyber-threats, such as Denial-of-Service (DoS) attacks, that can adversely impact the operation of the power grid. Researchers have proposed using Software Defined Network frameworks to enhance cybersecurity of the smart grid, but there is a lack of benchmarking and comparative analyses among the many techniques. In this work, a distributed three-controller software-defined networking (D3-SDN) architecture, benchmarking, and comparative analysis with other techniques is presented. The selected distributed flat SDN architecture divides the network horizontally into multiple areas or clusters, where each cluster is handled by a single Open Network Operating System (ONOS) controller. A case study using the IEEE 118-bus system is provided to compare the performance of the presented ONOS-managed D3-SDN, against the POX controller. In addition, the proposed architecture outperforms a single SDN controller framework by a tenfold increase in throughput; a reduction in latency of > 20%; and an increase in throughput of approximately 11% during the DoS attack scenarios.

Agnew Jr., Dennis↗

Development, Testing and Validation of a Waste Assay System for the Measurement and Characterisation of Active Spent Fuel Element Debris From UK Magnox Reactors - 12533

This paper describes the development, testing and validation of a waste measurement instrument for characterising active remote handled radioactive waste arising from the operation of Magnox reactors in the United Kingdom. Following operation in UK Magnox gas cooled reactors and a subsequent period of cooling, parts of the magnesium-aluminium alloy cladding were removed from spent fuel and the uranium fuel rods with the remaining cladding were removed to Sellafield for treatment. The resultant Magnox based spent fuel element debris (FED), which constitutes active intermediate level waste (ILW) has been stored in concrete vaults at the reactor sites. As part of the decommissioning of the FED vaults the FED must be removed, measured and characterised and placed in intermediate storage containers. The present system was developed for use at the Trawsfynydd nuclear power station (NPS), which is in the decommissioning phase, but the approach is potentially applicable to FED characterisation at all of the Magnox reactors. The measurement system consists of a heavily shielded and collimated high purity Germanium (HPGe) detector with electromechanical cooling and a high count-rate preamplifier and digital multichannel pulse height analyser. The HPGe based detector system is controlled by a software code, which stores the measurement result and allows a comprehensive analysis of the measured FED data. Fuel element debris is removed from the vault and placed on a tray to a uniform depth of typically 10 cm for measurement. The tray is positioned approximately 1.2 meters above the detector which views the FED through a tungsten collimator with an inverted pyramid shape. At other Magnox sites the positions may be reversed with the shielded and collimated HPGe detector located above the tray on which the FED is measured. A comprehensive Monte Carlo modelling and analysis of the measurement process has been performed in order to optimise the measurement geometry and eliminate interferences from radioactive sources and FED in the immediate vicinity of the measurement position. The detector system has been calibrated and high activity radioactive sources of Cs-137, Co-60 and Na-22 have been used to validate the measurement process. The data acquisition and analysis software code has been tested and validated in keeping with the software quality assurance requirements of both ISO:9001-2008 - TICK-IT in the UK and NQA-1. The measurement and analysis system has been comprehensively tested with high activity sources, is flexible and may be applicable to a wide range of remote handled radioactive waste measurement applications. It is due to be installed at Trawsfynydd NPS later this year. This paper describes the Waste Tray Assay System (WTAS) that has been developed for the measurement of Magnox FED waste. The WTAS has been tested with a range of radioactive sources and its operation has been simulated with benchmarked MCNP Monte Carlo calculations. The measurement software has been validated as has the operation of the system for a range of strong radioactive sources. A system based on the design is due for installation and operation in 2012. The system has application to the measurement of Magnox Fuel Element Debris (FED) waste at other Magnox reactor sites. The major design objective of the WTAS that has been achieved is the ability of the assay system to determine the content of Cs-137, and in turn to enable the fissile burden to be assessed using a radionuclide fingerprint, in the presence of higher and highly variable quantities of Co-60, typically from nimonic springs. The approach can be used in other Magnox FED waste configurations where the detector is located above the FED waste sorting tray and where the collimation is fixed below the detector and at a distance above the tray. In this case, which has also been investigated, there are different shielding problems and mechanical support issues. The extensive use of MCNP Monte Carlo modelling to simulate the geometry of the sorting cell and the distribution of radioactive sources has helped to ensure that all of the detector shielding requirements are addressed and suitable Cs-137 and Co-60 discrimination can be achieved. The WTAS in its present form or in other configurations has relevance to the measurement of other active ILW and highly active RH waste. Examples include high activity RH LLW and RH TRU (Transuranic) waste as defined in the United States arising from both commercial nuclear and Department of Energy (DOE) operations. The analysis is able to analyse a range of radionuclides beyong those expected in the Magnox FED cases. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

TU-AB-BRC-10: Modeling of Radiotherapy Linac Source Terms Using ARCHER Monte Carlo Code: Performance Comparison of GPU and MIC Computing Accelerators

Purpose: (1) To perform phase space (PS) based source modeling for Tomotherapy and Varian TrueBeam 6 MV Linacs, (2) to examine the accuracy and performance of the ARCHER Monte Carlo code on a heterogeneous computing platform with Many Integrated Core coprocessors (MIC, aka Xeon Phi) and GPUs, and (3) to explore the software micro-optimization methods. Methods: The patient-specific source of Tomotherapy and Varian TrueBeam Linacs was modeled using the PS approach. For the helical Tomotherapy case, the PS data were calculated in our previous study (Su et al. 2014 41(7) Medical Physics). For the single-view Varian TrueBeam case, we analytically derived them from the raw patient-independent PS data in IAEA’s database, partial geometry information of the jaw and MLC as well as the fluence map. The phantom was generated from DICOM images. The Monte Carlo simulation was performed by ARCHER-MIC and GPU codes, which were benchmarked against a modified parallel DPM code. Software micro-optimization was systematically conducted, and was focused on SIMD vectorization of tight for-loops and data prefetch, with the ultimate goal of increasing 512-bit register utilization and reducing memory access latency. Results: Dose calculation was performed for two clinical cases, a Tomotherapy-based prostate cancer treatment and a TrueBeam-based left breast treatment. ARCHER was verified against the DPM code. The statistical uncertainty of the dose to the PTV was less than 1%. Using double-precision, the total wall time of the multithreaded CPU code on a X5650 CPU was 339 seconds for the Tomotherapy case and 131 seconds for the TrueBeam, while on 3 5110P MICs it was reduced to 79 and 59 seconds, respectively. The single-precision GPU code on a K40 GPU took 45 seconds for the Tomotherapy dose calculation. Conclusion: We have extended ARCHER, the MIC and GPU-based Monte Carlo dose engine to Tomotherapy and Truebeam dose calculations.

60 APPLIED LIFE SCIENCES↗

SolCalc: A Suite for the Calculation and the Display of Magnetic Fields Generated by Solenoid Systems

SolCalc is a software suite that computes and displays magnetic fields generated by a three dimensional (3D) solenoid system. Examples of such systems are the Mu2e magnet system and Helical Solenoids for muon cooling systems. SolCalc was originally coded in Matlab, and later upgraded to a compiled version (called MEX) to improve solving speed. Matlab was chosen because its graphical capabilities represent an attractive feature over other computer languages. Solenoid geometries can be created using any text editor or spread sheets and can be displayed dynamically in 3D. Fields are computed from any given list of coordinates. The field distribution on the surfaces of the coils can be displayed as well. SolCalc was benchmarked against a well-known commercial software for speed and accuracy and the results compared favorably.

43 PARTICLE ACCELERATORS↗

Flexible Reinforcement Learning Framework for Building Control using EnergyPlus-Modelica Energy Models

In recent years, reinforcement learning (RL) methods have been greatly enhanced by leveraging deep learning approaches. RL methods applied to building control have shown potential in many applications due to their ability to complement or replace conventional methods such as model-based or rule-based controls. However, RL-based building control software is likely tailored either to one target building system or to a specific RL method so that significant additional effort would be required to customize the RL-based controller for use in other building systems or with other RL approaches. Also, RL-based building controls usually depend on building energy simulations to train controllers, so emulating building dynamics (i.e., thermal dynamics and control dynamics) and capturing sub-hourly dynamic profiles are crucial to further the development of effective RL-based building control methods. To address these challenges, we present an open source RL-based control software employing a high-fidelity hybrid EnergyPlus-Modelica building energy model which emulates building dynamics at 1-minute resolution. This software consists of decoupled components (environment, building emulator, control agent, and RL algorithm), which allows for quick prototyping and benchmarking of standard RL algorithms in different systems; for example, a single component can be replaced without revising all of the software. To demonstrate this software framework, we conducted a benchmark study using an EnergyPlus-Modelica building energy model for a Chicago office building with an RL-based controller to dynamically control the chilled water temperature setpoint and the air handling unit supply air temperature setpoint on selected floors.

Lee, Joon-Yong↗

Fundamental limit of jet tagging

Identifying the origin of high-energy hadronic jets (jet tagging) has been a critical benchmark problem for machine learning in particle physics. Jets are ubiquitous at colliders and are complex objects that serve as prototypical examples of collections of particles to be categorized. Over the last decade, machine learning-based classifiers have replaced classical observables as the state of the art in jet tagging. Increasingly complex machine learning models are leading to increasingly more effective tagger performance. Our goal is to address the question of convergence—are we getting close to the fundamental limit on jet tagging or is there still potential for computational, statistical, and physical insights for further improvements? We address this question using state-of-the-art generative models to create a realistic, synthetic dataset with a known jet tagging optimum. Various state-of-the-art taggers are deployed on this dataset, showing that there is a significant gap between their performance and the optimum. Our dataset and software are made public to provide a benchmark task for future developments in jet tagging and other areas of particle physics.

Artificial intelligence↗

Preparatory Verification of a New Monte Carlo Code cosRMC based on VERA Core Physics Benchmark

To promote nuclear power plant system design and simulation software self-reliance development, DOE of China approved a R and D key project to develop an integrated engine for pressurized water reactor core and system design and analysis, referred as COSINE hereafter. The scope of this code package includes reactor physics, thermal hydraulics, safety analysis, severe accidents, probability safety assessment and etc. Many universities, research institutes and nuclear related corporations in China are involved in the code development, among which the State Nuclear Power Software Development Center (SNPSDC) plays the leading role. With the requirement of accurate three-dimensional modeling of the advanced, special or new complex core in reactor neutronics and shielding analysis, Monte Carlo method is becoming a more powerful tool for reactor analysis. Thus, a new Monte Carlo transport code named cosRMC (abbreviation for COSINE Reactor Monte Carlo) has been developed jointly by SNPSDC and Department of Engineering Physics, Tsinghua University, the main developer of RMC code. The cosRMC now has many functions as criticality calculation, fixed-source calculation, burnup calculation, and etc. Some techniques for geometry treatment, new burnup algorithm, source convergence acceleration, massive tally and parallel calculation have been implemented in cosRMC to improve the efficiency. In order to ensure its reliability, several types of V and V work have been and are being carried out for cosRMC. In this paper, some preparatory verification results based on VERA Core Physics Benchmark are presented. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Measurement of Turbulent Flow Phenomena for the Lower Plenum of a Prismatic Gas-Cooled Reactor

Mean velocity field and turbulence data are presented that measure turbulent flow phenomena in an approximately 1:7 scale model of a region of the lower plenum of a typical prismatic gas-cooled reactor (GCR) similar to a General Atomics design (Gas-Turbine-Modular Helium Reactor). The datawere obtained in the Matched-Index-of-Refraction (MIR) facility at Idaho National Laboratory (INL) and are offered as a benchmark for assessing computational fluid dynamics (CFD) software. This experiment has been selected as the first Standard Problem endorsed by the Generation IV International Forum. The primary objective of this paper is to document the experiment and present a sample of the data set that has been established for this standard problem. Present results concentrate on the region of the lower plenum near its far reflector wall (away from the outlet duct). The flowin the lower plenum consists of multiple jets injected into a confined crossflow—with obstructions. The model consists of a row of full circular posts along its centerline with half-posts on the two parallel walls to approximate flow scaled to that expected from the staggered parallel rows of posts in the reactor design. Posts, side walls and end walls are fabricated from clear, fused quartz to match the refractive index of the mineral oil working fluid so that optical techniques may be employed for the measurements. The benefit of the MIR technique is that it permits optical measurements to determine flow characteristics in complex passages and around objects to be obtained without locating intrusive transducers that will disturb the flow field and without distortion of the optical paths. An advantage of the INL system is its large size, leading to improved spatial and temporal resolution compared to similar facilities at smaller scales. A three-dimensional (3D) particle image velocimetry (PIV) system was used to collect the data. Inlet-jet Reynolds numbers (based on the hydraulic diameter of the jet and the timemean average flow rate) are approximately 4300 and 12,400. Uncertainty analysis and a discussion of the standard problem are included. The measurements reveal complicated flow patterns that include several large recirculation zones, reverse flow near the simulated reflector wall, recirculation zones in the upper portion of the plenum and complex flow patterns around the support posts. Data include three-dimensional PIV images of flow planes, data displays along the coordinate planes (slices) and presentations that describe the component flows at specific regions in the model.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Benchmarking the Performance of Neuromorphic and Spiking Neural Network Simulators

Software simulators play a critical role in the development of new algorithms and system architectures in any field of engineering. Neuromorphic computing, which has shown potential in building brain-inspired energy-efficient hardware, suffers a slow-down in the development cycle due to a lack of flexible and easy-to-use simulators of either neuromorphic hardware itself or of spiking neural networks (SNNs), the type of neural network computation executed on most neuromorphic systems. While there are several openly available neuromorphic or SNN simulation packages developed by a variety of research groups, they have mostly targeted computational neuroscience simulations, and only a few have targeted small-scale machine learning tasks with SNNs. Evaluations or comparisons of these simulators have often targeted computational neuroscience-style workloads. In this work, we seek to evaluate the performance of several publicly available SNN simulators with respect to non-computational neuroscience workloads, in terms of speed, flexibility, and scalability. We evaluate the performance of the NEST, Brian2, Brian2GeNN, BindsNET and Nengo packages under a common front-end neuromorphic framework. Our evaluation tasks include a variety of different network architectures and workload types to mimic the computation common in different algorithms, including feed-forward network inference, genetic algorithms, and reservoir computing. We also study the scalability of each of these simulators when running on different computing hardware, from single core CPU workstations to multi-node supercomputers. Our results show that the BindsNET simulator has the best speed and scalability for most of the SNN workloads (sparse, dense, and layered SNN architectures) on a single core CPU. However, when comparing the simulators leveraging the GPU capabilities, Brian2GeNN outperforms the others for these workloads in terms of scalability. NEST performs the best for small sparse networks and is also the most flexible simulator in terms of reconfiguration capability NEST shows a speedup of at least 2x compared to the other packages when running evolutionary algorithms for SNNs. The multi-node and multi-thread capabilities of NEST show at least 2x speedup compared to the rest of the simulators (single core CPU or GPU based simulators) for large and sparse networks. We conclude our work by providing a set of recommendations on the suitability of employing these simulators for different tasks and scales of operations. We also present the characteristics for a future generic ideal SNN simulator for different neuromorphic computing workloads.

97 MATHEMATICS AND COMPUTING↗