Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmarking Software”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

Reassessment of PHDS Fulcrum40h High Purity Germanium (HPGe) Detector System Performance

This report presents a comprehensive evaluation of the updated PHDS Fulcrum40h High Purity Germanium (HPGe) detector system, benchmarking its performance, usability, and software capabilities against the current National Nuclear Security Administration (NNSA) Nuclear Emergency Support Team (NEST) HPGe Detector System Requirements Document. Systematic measurements were conducted using a single Fulcrum40h detector to assess key parameters including gamma efficiency and resolution, neutron detection efficiency, gamma pulse-pileup response, and gamma-to-neutron crosstalk. The Fulcrum40h system, acquired in August 2022, has undergone recent updates by PHDS to address deficiencies identified following the initial NEST requirements release. The results provide critical insights into the detector’s operational capabilities and compliance with NNSA NEST standards.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Technology Transfer Plan: LuNaMaps Project

The main contribution of this project is the combined knowledge of terrain relative navigation experts and lunar scientists who are familiar with both the lunar orbital imagery and the instruments that collected the data as well as how a TRN system utilizes map data. This knowledge comes in the form of published technical papers, benchmark map data sets, and software tools that can help others automate the process of creating the necessary maps for their own landing sites in the future. This document represents the project's plans to share all the lessons learned, processes developed, and applicable software tools with the public.

optical navigation↗

Tutorial on LuNaMaps Developed Tools andProcesses for Mapping the Lunar Surface

The main contribution of this project is the combined knowledge of terrain relative navigation experts and lunar scientists who are familiar with both the lunar orbital imagery and the instruments that collected the data as well as how a TRN system utilizes map data. This knowledge comes in the form of published technical papers, benchmark map data sets, and software tools that can help others automate the process of creating the necessary maps for their own landing sites in the future. This presentation provides a brief overview of the tools and processes developed by the project.

optical navigation↗

Sensitivity-based Similarity Metrics for New Experiment Design Optimization

The nuclear data used in advanced reactor simulations requires validation. Data from nuclear criticality experiments can provide this validation. New nuclear criticality experiment design requires extensive knowledge and expert judgement such that the experimental design parameters are selected in such a way to keep the experiment subcritical. To aide in this experimental design process, professionals can utilize sensitivity and uncertainty analysis. Sensitivity and uncertainty analysis relies on matching new application experiments with currently existing benchmark experiments. Currently, there is functionality in the Whisper 1.1 software package to calculate a similarity metric based on neutron multiplication factor sensitivity coefficients between a new application designed by the user and existing International Criticality Safety Benchmark Experiment Project (ICSBEP) benchmarks. The Whisper 1.1 software package is included in Monte Carlo N-Particle ® Code Version 6.21 (MCNP ® 6.2). This work is geared toward expanding this capability to new similarity metrics based on beta-effective sensitivity coefficients and reactivity coefficient sensitivity coefficients. While the investigation of these sensitivity coefficients is presented in detail in separate works at this same conference, this work will be primarily focused on studying the similarity metrics in more detail. These similarity metrics will then be incorporated into the optimization algorithms used for experiment design in EUCLID (Experiments Underpinned by Computational Learning for Improvements in nuclear Data), which is a Los Alamos National Laboratory (LANL) project designed to constrain nuclear data of interest, such that adjustments can be made to possible inaccuracies. A more detailed optimization can be subsequently performed by breaking down these similarity metrics by isotope, reaction, and energy.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Multiresolution molecular mechanics: Implementation and efficiency

Atomistic/continuum coupling methods combine accurate atomistic methods and efficient continuum methods to simulate the behavior of highly ordered crystalline systems. Coupled methods utilize the advantages of both approaches to simulate systems at a lower computational cost, while retaining the accuracy associated with atomistic methods. Many concurrent atomistic/continuum coupling methods have been proposed in the past; however, their true computational efficiency has not been demonstrated. The present work presents an efficient implementation of a concurrent coupling method called the Multiresolution Molecular Mechanics (MMM) for serial, parallel, and adaptive analysis. First, we present the features of the software implemented along with the associated technologies. The scalability of the software implementation is demonstrated, and the competing effects of multiscale modeling and parallelization are discussed. Then, the algorithms contributing to the efficiency of the software are presented. These include algorithms for eliminating latent ghost atoms from calculations and measurement-based dynamic balancing of parallel workload. The efficiency improvements made by these algorithms are demonstrated by benchmark tests. The efficiency of the software is found to be on par with LAMMPS, a state-of-the-art Molecular Dynamics (MD) simulation code, when performing full atomistic simulations. Speed-up of the MMM method is shown to be directly proportional to the reduction of the number of the atoms visited in force computation. Finally, an adaptive MMM analysis on a nanoindentation problem, containing over a million atoms, is performed, yielding an improvement of 6.3–8.5 times in efficiency, over the full atomistic MD method. For the first time, the efficiency of a concurrent atomistic/continuum coupling method is comprehensively investigated and demonstrated.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

Defect measurement and analysis of JPL ground software: a case study

Ground software systems at JPL must meet high assurance standards while remaining on schedule due to relatively immovable launch dates for spacecraft that will be controlled by such systems. Toward this end, the Software Quality Improvement (SQI) project's Measurement and Benchmarking (M&B) team is collecting and analyzing defect data of JPL ground system software projects to build software defect prediction models. The aim of these models is to improve predictability with regard to software quality activities. Predictive models will quantitatively define typical trends for JPL ground systems as well as Critical Discriminators (CDs) to provide explanations for atypical deviations from the norm at JPL. CDs are software characteristics that can be estimated or foreseen early in a software project's planning. Thus, these CDs will assist in planning for the predicted degree to which software quality activities for a project are likely to deviation from the normal JPL ground system based on pasted experience across the lab.

defects↗

Intern-Artificial Intelligence Benchmarking

Benchmarks provide a standardized method for evaluating different AI models, enabling reproducibility and comparison between models, and facilitating scientific progress. As AI models continue to develop rapidly, incorporating new datasets, capabilities, and architectures becomes more complicated. Therefore, the current static benchmarks become increasingly irrelevant. The MLCommons team argues that to make AI benchmarks more relevant, it involves making the benchmarks themselves more dynamic, as well as technical innovations that make it easier for scientists and researchers at all levels to use and contribute to the benchmarks. The current progress in technical innovation is a software that allows for a detailed view of a collection of AI benchmarks to be output in various formats that are easily readable and accessible.

Krishnan, Anjay [Fermilab]↗

Scientific Data Management Integrated Software Infrastructure Center

This work provides software that enables scientific applications to more efficiently access available storage resources at different levels of interfaces. We developed scalable techniques and optimizations for PVFS parallel file systems, MPI I/O, and parallel netCDF I/O library. These implementations were evaluated using production application I/O kernels as well as popular I/O benchmarks and demonstrated promising results. The software developed under this work has been made available to the public via MCS, ANL web sites.

97 MATHEMATICS AND COMPUTING↗

Enabling High Performance Application I/O

This work provides software that enables scientific applications to more efficiently access available storage resources at different levels of interfaces. We developed scalable techniques and optimizations for PVFS parallel file systems, MPI I/O, and parallel netCDF I/O library. These implementations were evaluated using production application I/O kernels as well as popular I/O benchmarks and demonstrated promising results. The software developed under this work has been made available to the public via MCS, ANL web sites.

97 MATHEMATICS AND COMPUTING↗

A Method for Selecting Software for Dynamic Event Analysis II: the Taylor Anvil and Dynamic Brazilian Tests

New nuclear power reactor designs will require resistance to a variety of possible malevolent attacks as well as traditional dynamic accident scenarios. The design/analysis team may be faced with a broad range of phenomena including air and ground blasts, high-velocity penetrators or shaped charges, and vehicle or aircraft impacts. With a host of software tools available to address these high-energy events, the analysis team must evaluate and select the software most appropriate for their particular set of problems. The accuracy of the selected software should then be validated with respect to the phenomena governing the interaction of the threat and structure. Several software codes are available for the study of blast, impact, and other shock phenomena. At the Idaho National Laboratory (INL), a study is underway to investigate the comparative characteristics of a group of shock and high-strain rate physics codes including ABAQUS, LS-DYNA, CTH, ALEGRA, and ALE-3D. In part I of this report, a series of five benchmark problems to exercise some important capabilities of the subject software was identified. The benchmark problems selected are a Taylor cylinder test, a split Hopkinson pressure bar test, a free air blast, the dynamic splitting tension (Brazilian) test, and projectile penetration of a concrete slab. Part II-- this paper-- reports the results of two of the benchmark problems: the Taylor cylinder and the dynamic Brazilian test. The Taylor cylinder test is a method to determine the dynamic yield properties of materials. The test specimen is a right circular cylinder which is impacted against a theoretically rigid target. The cylinder deforms upon impact, with the final shape depending upon the dynamic yield stress, in turn a function of strain and strain rate. The splitting tension test, or Brazilian test, is a method to measure the tensile strength of concrete using a cylindrical specimen. The specimen is loaded diametrically in compression, producing a fracture at the center of the specimen that propagates toward the loading points until the cylinder is split. To generate a dynamic load, different methods such as a drop-weight or a split Hopkinson pressure bar are employed. The Taylor anvil and dynamic Brazilian test analyses are presented, including discussion of the analysis approach for each of the five subject software packages; comparison of results both among the codes and to physical test results; and conclusions as to the applicability of the subject codes to these two problems. Studies of the remaining three benchmark problems and overall conclusions will be presented in future reports.

42 ENGINEERING↗

FireHose Streaming Benchmarks

The FireHose Streaming Benchmarks are a suite of stream-processing benchmarks defined to enable comparison of streaming software and hardware, both quantitatively vis-a-vis the rate at which they can process data, and qualitatively by judging the effort involved to implement and run the benchmarks. Each benchmark has two parts. The first is a generator which produces and outputs datums at a high rate in a specific format. The second is an analytic which reads the stream of datums and is required to perform a well-defined calculation on the collection of datums, typically to find anomalous datums that have been created in the stream by the generator. The FireHose suite provides code for the generators, sample code for the analytics (which users are free to re-implement in their own custom frameworks), and a precise definition of each benchmark calculation.

Plimpton, Steve↗

FireHose Streaming Benchmarks

The FireHose Streaming Benchmarks are a suite of stream-processing benchmarks defined to enable comparison of streaming software and hardware, both quantitatively vis-a-vis the rate at which they can process data, and qualitatively by judging the effort involved to implement and run the benchmarks. Each benchmark has two parts. The first is a generator which produces and outputs datums at a high rate in a specific format. The second is an analytic which reads the stream of datums and is required to perform a well-defined calculation on the collection of datums, typically to find anomalous datums that have been created in the stream by the generator. The FireHose suite provides code for the generators, sample code for the analytics (which users are free to re-implement in their own custom frameworks), and a precise definition of each benchmark calculation.

Karl Anderson, Steve Plimpton↗

Building America Research Benchmark Definition, Updated December 19, 2008

To track progress toward aggressive multi-year, whole-house energy savings goals of 40%–70% and onsite power production of up to 30%, the U.S. Department of Energy (DOE) Residential Buildings Program and the National Renewable Energy Laboratory (NREL) developed the Building America (BA) Research Benchmark in consultation with the BA industry teams. The Benchmark is generally consistent with mid-1990s standard practice, as reflected in the Home Energy Rating System (HERS) Technical Guidelines (RESNET 2002), with additional definitions that allow the analyst to evaluate all residential end-uses, an extension of the traditional HERS rating approach that focuses on space conditioning and hot water. Unlike the reference homes used for HERS, ENERGY STAR ® , and most energy codes, the Benchmark represents typical construction at a fixed point in time so it can be used as the basis for Building America’s multi-year energy savings goals without the complication of chasing a “moving target.” As time passes, we expect energy codes to become increasingly energy efficient compared to the Benchmark as better construction practices and more efficient equipment become commonplace in the market. A series of user profiles, intended to represent the behavior of a “standard” set of occupants, was created for use in conjunction with the Benchmark. The Benchmark is intended for use with detached and attached single-family housing only. It is not suitable for multi-family housing as written. Energy analysis of a Prototype compared to the Benchmark can be performed with any software tool that complies with the BA Performance Analysis Procedures (Hendron et al. 2004). In addition, NREL will provide examples of technology packages that can be used to achieve different source energy savings based on BEopt analysis results (Anderson and Roberts, 2008). These technology packages, or alternative packages that provide equivalent source energy savings, may be used to demonstrate minimum whole house source energy savings for BA Gate reviews.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

nbodykit: An Open-source, Massively Parallel Toolkit for Large-scale Structure

We present nbodykit, an open-source, massively parallel Python toolkit for analyzing large-scale structure (LSS) data. Using Python bindings of the Message Passing Interface, we provide parallel implementations of many commonly used algorithms in LSS. nbodykit is both an interactive and scalable piece of scientific software, performing well in a supercomputing environment while still taking advantage of the interactive tools provided by the Python ecosystem. Existing functionality includes estimators of the power spectrum, two- and three-point correlation functions, a friends-of-friends grouping algorithm, mock catalog creation via the halo occupation distribution technique, and approximate N-body simulations via the FastPM scheme. The package also provides a set of distributed data containers, insulated from the algorithms themselves, that enables nbodykit to provide a unified treatment of both simulation and observational data sets. nbodykit can be easily deployed in a high-performance computing environment, overcoming some of the traditional difficulties of using Python on supercomputers. We provide performance benchmarks illustrating the scalability of the software. The modular, component-based approach of nbodykit allows researchers to easily build complex applications using its tools. The package is extensively documented at http://nbodykit.readthedocs.io, which also includes an interactive set of example recipes for new users to explore. As open-source software, we hope nbodykit provides a common framework for the community to use and develop in confronting the analysis challenges of future LSS surveys.

79 ASTRONOMY AND ASTROPHYSICS↗

The Arctic-Boreal vulnerability experiment model benchmarking system

Abstract NASA’s Arctic-Boreal Vulnerability Experiment (ABoVE) integrates field and airborne data into modeling and synthesis activities for understanding Arctic and Boreal ecosystem dynamics. The ABoVE Benchmarking System (ABS) is an operational software package to evaluate terrestrial biosphere models against key indicators of Arctic and Boreal ecosystem dynamics, i.e.: carbon biogeochemistry, vegetation, permafrost, hydrology, and disturbance. The ABS utilizes satellite remote sensing data, airborne data, and field data from ABoVE as well as collaborating research networks in the region, e.g.: the Permafrost Carbon Network, the International Soil Carbon Network, the Northern Circumpolar Soil Carbon Database, AmeriFlux sites, the Moderate Resolution Imaging Spectroradiometer, the Orbiting Carbon Observatory 2, and the Soil Moisture Active Passive mission. The ABS is designed to be interactive for researchers interested in having their models accurately represent observations of key Arctic indicators: a user submits model results to the system, the system evaluates the model results against a set of Arctic-Boreal benchmarks outlined in the ABoVE Concise Experiment Plan, and the user then receives a quantitative scoring of model strengths and deficiencies through a web interface. This interactivity allows model developers to iteratively improve their model for the Arctic-Boreal Region by evaluating results from successive model versions. We show here, for illustration, the improvement of the Lund–Potsdam–Jena-Wald Schnee und Landschaft (LPJwsl) version model through the ABoVE ABS as a new permafrost module is coupled to the existing model framework. The ABS will continue to incorporate new benchmarks that address indicators of Arctic-Boreal ecosystem dynamics as they become available.

Stofferahn, Eric (ORCID:0000000169604193)↗

The Arctic-Boreal vulnerability experiment model benchmarking system

NASA's Arctic-Boreal Vulnerability Experiment (ABoVE) integrates field and airborne data into modeling and synthesis activities for understanding Arctic and Boreal ecosystem dynamics. The ABoVE Benchmarking System (ABS) is an operational software package to evaluate terrestrial biosphere models against key indicators of Arctic and Boreal ecosystem dynamics, i.e.: carbon biogeochemistry, vegetation, permafrost, hydrology, and disturbance. The ABS utilizes satellite remote sensing data, airborne data, and field data from ABoVE as well as collaborating research networks in the region, e.g.: the Permafrost Carbon Network, the International Soil Carbon Network, the Northern Circumpolar Soil Carbon Database, AmeriFlux sites, the Moderate Resolution Imaging Spectroradiometer, the Orbiting Carbon Observatory 2, and the Soil Moisture Active Passive mission. The ABS is designed to be interactive for researchers interested in having their models accurately represent observations of key Arctic indicators: a user submits model results to the system, the system evaluates the model results against a set of Arctic-Boreal benchmarks outlined in the ABoVE Concise Experiment Plan, and the user then receives a quantitative scoring of model strengths and deficiencies through a web interface. This interactivity allows model developers to iteratively improve their model for the Arctic-Boreal Region by evaluating results from successive model versions. We show here, for illustration, the improvement of the Lund–Potsdam–Jena-Wald Schnee und Landschaft (LPJwsl) version model through the ABoVE ABS as a new permafrost module is coupled to the existing model framework. The ABS will continue to incorporate new benchmarks that address indicators of Arctic-Boreal ecosystem dynamics as they become available.

ABoVE↗

Identification of ultramodified proteins using top-down spectra

Post-translational modifications (PTMs) play an important role in various biological processes through changing protein structure and function. Some ultramodified proteins (like histones) have multiple PTMs forming PTM patterns that define the functionality of a protein. While bottom-up mass spectrometry (MS) has been successful in identifying individual PTMs within short peptides, it is unable to identify PTM patterns spread along entire proteins in a coordinated fashion. In contrast, top-down MS analyzes intact proteins and reveals PTM patterns along the entire proteins. However, while recent advances in instrumentation have made top-down MS accessible to many laboratories, most computational tools for top-down MS focus on proteins with few PTMs and are unable to identify complex PTM patterns. We propose a new algorithm, MS-Align-E, that identifies both expected and unexpected PTMs in ultramodified proteins. We demonstrate that MS-Align-E identifies many protein forms of histone H4 and benchmark it against the currently accepted software tools.

Environmental Molecular Sciences Laboratory↗

Identification of Ultramodified Proteins Using Top-Down Mass Spectra

Post-translational modifications (PTMs) play an important role in various biological processes through changing protein structure and function. Some ultramodified proteins (like histones) have multiple PTMs forming PTM patterns that define the functionality of a protein. While bottom-up mass spectrometry (MS) has been successful in identifying individual PTMs within short peptides, it is unable to identify PTM patterns spread along entire proteins in a coordinated fashion. In contrast, top-down MS analyzes intact proteins and reveals PTM patterns along the entire proteins. However, while recent advances in instrumentation have made top-down MS accessible to many laboratories, most computational tools for top-down MS focus on proteins with few PTMs and are unable to identify complex PTM patterns. We propose a new algorithm, MS-Align-E, that identifies both expected and unexpected PTMs in ultramodified proteins. We demonstrate that MS-Align-E identifies many protein forms of histone H4 and benchmark it against the currently accepted software tools.

Environmental Molecular Sciences Laboratory↗