Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmarking Software”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Comparing Server Energy Use and Efficiency Using Small Sample Sizes

This report documents a demonstration that compared the energy consumption and efficiency of a limited sample size of server-type IT equipment from different manufacturers by measuring power at the server power supply power cords. The results are specific to the equipment and methods used. However, it is hoped that those responsible for IT equipment selection can used the methods described to choose models that optimize energy use efficiency. The demonstration was conducted in a data center at Lawrence Berkeley National Laboratory in Berkeley, California. It was performed with five servers of similar mechanical and electronic specifications; three from Intel and one each from Dell and Supermicro. Server IT equipment is constructed using commodity components, server manufacturer-designed assemblies, and control systems. Server compute efficiency is constrained by the commodity component specifications and integration requirements. The design freedom, outside of the commodity component constraints, provides room for the manufacturer to offer a product with competitive efficiency that meets market needs at a compelling price. A goal of the demonstration was to compare and quantify the server efficiency for three different brands. The efficiency is defined as the average compute rate (computations per unit of time) divided by the average energy consumption rate. The research team used an industry standard benchmark software package to provide a repeatable software load to obtain the compute rate and provide a variety of power consumption levels. Energy use when the servers were in an idle state (not providing computing work) were also measured. At high server compute loads, all brands, using the same key components (processors and memory), had similar results; therefore, from these results, it could not be concluded that one brand is more efficient than the other brands. The test results show that the power consumption variability caused by the key components as a group is similar to all other components as a group. However, some differences were observed. The Supermicro server used 27 percent more power at idle compared to the other brands. The Intel server had a power supply control feature called cold redundancy, and the data suggest that cold redundancy can provide energy savings at low power levels. Test and evaluation methods that might be used by others having limited resources for IT equipment evaluation are explained in the report.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

NOSS Altimeter Detailed Algorithm specifications

The details of the algorithms and data sets required for satellite radar altimeter data processing are documented in a form suitable for (1) development of the benchmark software and (2) coding the operational software. The algorithms reported in detail are those established for altimeter processing. The algorithms which required some additional development before documenting for production were only scoped. The algorithms are divided into two levels of processing. The first level converts the data to engineering units and applies corrections for instrument variations. The second level provides geophysical measurements derived from altimeter parameters for oceanographic users.

Hancock, D. W.↗

A small evaluation suite for Ada compilers

After completing a small Ada pilot project (OCC simulator) for the Multi Satellite Operations Control Center (MSOCC) at Goddard last year, the use of Ada to develop OCCs was recommended. To help MSOCC transition toward Ada, a suite of about 100 evaluation programs was developed which can be used to assess Ada compilers. These programs compare the overall quality of the compilation system, compare the relative efficiencies of the compilers and the environments in which they work, and compare the size and execution speed of generated machine code. Another goal of the benchmark software was to provide MSOCC system developers with rough timing estimates for the purpose of predicting performance of future systems written in Ada.

Wilke, Randy↗

Software Requirements Document: Benchmarking Algorithm for RadioNuclide Identification (BARNI)

This document provides the requirements for the Benchmark Algorithm for Radio-Nuclide Identification (BARNI) software, an open-source tool for comparing radionuclide identification algorithm performance. It specifies the requirements for the user interface, maintainability, documentation and testing. In addition, this document outlines the core functionality of the product, which will align with the needs of the Office of Nuclear Smuggling Detection and Deterrence (NSDD). Specifically, these include interoperability with RASE, and compatibility with the FLIR R400.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Critical Assessment of Metagenome Interpretation—a benchmark of metagenomics software

Methods for assembly, taxonomic profiling and binning are key to interpreting metagenome data, but a lack of consensus about benchmarking complicates performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on highly complex and realistic data sets, generated from ~700 newly sequenced microorganisms and ~600 novel viruses and plasmids and representing common experimental setups. Assembly and genome binning programs performed well for species represented by individual genomes but were substantially affected by the presence of related strains. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below family level. Parameter settings markedly affected performance, underscoring their importance for program reproducibility. The CAMI results highlight current challenges but also provide a roadmap for software selection to answer specific research questions.

59 BASIC BIOLOGICAL SCIENCES↗

Initial Validation of the Neutron Physics Codes in COSINE Using the MIT BEAVRS Benchmark

A software package for nuclear power plant design and analysis, known as COSINE, has been developed by State Nuclear Power Software Development Center (SNPSDC). A detailed benchmark called BEAVRS which includes measured reactor data is presented by MIT. The BEAVRS benchmark is based on a commercial power plant and provides a highly-detailed Pressurized Water Reactor (PWR) specification with measured operational data that has been used to validate the neutron physics codes in COSINE. In this paper, a COSINE core model is prepared to calculate the BEAVRS benchmark. The first stage validation is focused on the depletion of two cycles and initial physics test of cycle 1. The calculation results agree well with the measured data. It could be confirmed that the COSINE codes package has the ability to design and analyze a commercial PWR core, and the precision of the calculation results can meet the requirements of industrial design and analysis. A calculation of MIT BEAVRS benchmark using COSINE has been performed in this paper. A lot of measurements are given in the benchmark specification which can be used for code validation. The calculation includes a depletion calculation of both two cycles and a calculation of the initial physics test of cycle 1. COSINE neutron physics codes use the mature industrial calculation method which can achieve reasonable results with acceptable accuracy and speed. Through the calculation, it can be seen that the difference of critical boron concentration calculated from cos Nu is less than 25 pcm compared with measured data. And the relative difference of rod worth is less than 10%. The isothermal temperature coefficient difference between measurement and calculation is less than 1 pcm/deg. F. All results meet the criteria of ANS. The validation for measurements after initial physics test following a more complex power history and the validation for in-core detector signals are underway. (authors)

97 MATHEMATICS AND COMPUTING↗

Software Requirement Document: Benchmarking Algorithms for RadioNuclide Identification (BARNI)

The Department of Energy (DOE) Office of Nuclear Smuggling Detection and Deterrence (NSDD) seeks to improve the performance of RadioIsotope Identification Devices (RIIDs). RIIDs enable users to identifying threat nuclides in applications such as secondary search, facility inspection, and emergency response. As part of this effort NSDD has developed Replicative Assessment of Spectroscopic Equipment (RASE), an open-source tool which generates synthetic spectra and analyzes them with vendor’s radionuclide identification software in order to evaluate performance.

61 RADIATION PROTECTION AND DOSIMETRY↗

SU-C-202-05: Pilot Study of Online Treatment Evaluation and Adaptive Re-Planning for Laryngeal SBRT

Purpose: We have instigated a phase I trial of 5-fraction stereotactic body radiotherapy (SBRT) for advanced-stage laryngeal cancer. We conducted this pilot dosimetric study to confirm the potential utility of online adaptive re-planning to preserve treatment quality. Methods: Ten cases of larynx cancer were evaluated. Baseline and daily SBRT treatment plans were generated per trial protocol. Daily volumetric images were acquired prior to every fraction of treatment. Reference simulation CT images were deformably registered to daily volumetric images using Eclipse. Planning contours were then deformably propagated to daily images. Reference SBRT plans were directly copied to calculate delivered dose distributions on deformed reference CT images. In-house software platform has been developed to calculate cumulative dose over a course of treatment in four steps: 1) deforming delivered dose grid to reference CT images using deformation information exported from Eclipse; 2) generating tetrahedrons using deformed dose grid as vertices; 3) resampling dose to a high resolution within every tetrahedron; 4) calculating dose-volume histograms. Our inhouse software was benchmarked with a commercial software, Mirada. Results: In all ten cases including 49 fractions of treatments, delivered daily doses were completely evaluated and treatment could be re-planned within 10 minutes. Prescription dose coverage of PTV was less than intended in 53% of fractions of treatment (mean: 94%, range: 84%–98%) while minimum coverage of CTV and GTV was 94% and 97%, respectively. Maximum bystander point dose limits to arytenoids, parotids, and spinal cord remained respected in all cases, although variances in carotid artery doses were observed in a minority of cases. Conclusion: Although GTV and CTV coverage is preserved by in-room 3D image guidance of larynx SBRT, PTV coverage can vary significantly from intended plans. Online adaptive treatment evaluation and re-planning is potentially necessary and our procedure is clinically applicable to fully preserve treatment quality. This project is supported by CPRIT Individual Investigator Research Award RP150386.

60 APPLIED LIFE SCIENCES↗

Quantum Application Specifications and Benchmarks

This software describes computational tasks for quantum computers that are derived from LANL basic science research applications such as the modeling of materials, chemicals and compounds at atomic scales. The computations focus on quantum simulation tasks, such as quantum dynamics, thermal state preparation and ground state estimation. The primary function of the software is to develop estimates of the requirements for large-scale fault-tolerant quantum computers to solve these scientific computations. The secondary focus of the software are codes for assessing the limitations of conducting quantum computations on classical computers.

Coffrin, Carleton↗

Benchmarking hypercube hardware and software

It was long a truism in computer systems design that balanced systems achieve the best performance. Message passing parallel processors are no different. To quantify the balance of a hypercube design, an experimental methodology was developed and the associated suite of benchmarks was applied to several existing hypercubes. The benchmark suite includes tests of both processor speed in the absence of internode communication and message transmission speed as a function of communication patterns.

Grunwald, Dirk C.↗

DOE FAIR Surrogate Benchmarks Supporting AI and Simulation Research (SBI Surrogate Benchmark Initiative) (Final Report)

Computational Science is being revolutionized by integrating AI and simulation and, in particular, by deep learning surrogate models that can replace all or part of traditional large‐scale HPC computations. Such surrogates can achieve remarkable performance improvements, as much as several orders of magnitude, and save both compute time and energy. The Surrogate Benchmark Initiative (SBI) project creates a community repository and FAIR (Findable, Accessible, Interoperable, and Reusable) data ecosystem for HPC application surrogate benchmarks. The SBI team comes from Argonne National Laboratory (ANL), Indiana University (IU), Rutgers University, the University of Tennessee, Knoxville (UTK), and the University of Virginia(UVA). SBI repositories include data, code, and all relevant collateral artifacts, that the science and engineering community needs to use and reuse these data sets and surrogates. SBI repositories generate active research from both participants in SBI and the broader AI and domain science communities. This project develops surrogates that use several different neural nets to learn and quickly infer the results of simulations and data systems and capture them as surrogate benchmarks with a rich set of metadata, covering. Data; Model; Metrics specification; Machine specification; Science, Speed, Power Results, We research FAIR metadata for these benchmarks. We develop application surrogate examples as benchmarks across many fields (ANL, UTK, IU, UVA). We also study non Surrogate benchmarks that have many common features and similar issues regarding FAIRness. We work with MLCommons (UVA, UTK), which is a major machine learning benchmarking activity where we get metadata ontologies, software, and benchmarks, benchmarks have datasets, models, and metadata, and they need a technical framework developed by UTK and Rutgers and deployed by UVA. We study features of Surrogates, including performance, training set size, and uncertainty quantification (Rutgers, UVA and IU).

97 MATHEMATICS AND COMPUTING↗

FAIR Surrogate Benchmarks Supporting AI and Simulation Research (Final Report)

Computational Science is being revolutionized by integrating AI and simulation and, in particular, by deep learning surrogate models that can replace all or part of traditional large‐scale HPC computations. Such surrogates can achieve remarkable performance improvements, as much as several orders of magnitude, and save both compute time and energy. The Surrogate Benchmark Initiative (SBI) project creates a community repository and FAIR (Findable, Accessible, Interoperable, and Reusable) data ecosystem for HPC application surrogate benchmarks. The SBI team comes from Argonne National Laboratory (ANL), Indiana University (IU), Rutgers University, the University of Tennessee, Knoxville (UTK), and the University of Virginia (UVA). SBI repositories include data, code, and all relevant collateral artifacts that the science and engineering community need to use and reuse these data sets and surrogates. SBI repositories generate active research from both the participants in SBI and the broad community of AI and domain scientists. This project develops surrogates that use several different neural nets to learn and quickly infer the results of simulations and data systems and captures them as surrogate benchmarks with a rich set of metadata covering: Data; Model; Metrics specification; Machine specification; and Science, Speed, and Power Results. We research FAIR metadata for these benchmarks. We develop application surrogate examples as benchmarks across many fields (ANL, UTK, IU, UVA). We also study non-Surrogate benchmarks that have many common features and similar issues as regards FAIRness. We work with MLCommons (UVA, UTK), which is a major machine learning benchmarking activity where we get metadata ontologies, software, and benchmarks, Benchmarks have datasets, models, and metadata and they need a technical framework developed by UTK and Rutgers and deployed by UVA. We study features of Surrogates including performance, training set size, and uncertainty quantification (Rutgers, UVA and IU).

97 MATHEMATICS AND COMPUTING↗

Towards Digital and Performance-Based Supervisory HVAC Control Delivery

Upgrading supervisory HVAC control in commercial buildings is one of the most attractive decarbonization tools at our disposal. Modern controls are software programs and can in theory be deployed at scale and with a low up-front carbon "pulse". In practice, however, control delivery is a disjointed and inefficient process, dominated by manual handoffs of imprecise English language documents. A particularly high barrier exists between control implementation and building energy modeling (BEM) which results in control sequences typically not being tested for correctness or performance before implementation. Together with industry partners, DOE and the national labs are developing an ecosystem of tools and standards that can support fully digital performance-based control delivery workflows. This paper describes this ecosystem, which consists of three mutually supportive efforts. Semantic models of buildings and their systems enable automatic configuration and installation of control software. Platform-neutral control descriptions separate control algorithms from control platforms and enable the creation of libraries of reference control implementations. Dynamic whole-building energy-control simulation that can execute physically realistic control sequences makes it possible to test and evaluate the performance of control sequences and then directly compile them for installation and execution in control systems. In addition to digitizing and streamlining project-level control delivery, these standards and related software support benchmarking of control algorithms, both rule-based and optimization-based, and help to both advance the state of the art and to implement ratings and programs that encourage the adoption of high-performance control.

building controls↗

Update to the Microcontroller Benchmark for Radiation Testing

LANL developed a benchmark of software code for radiation testing of microprocessors several years ago, and it was published under an open-source license on GitHub. Publishing the software is necessary for other researchers to adopt and implement this benchmark for radiation testing of other microprocessors to standardize test practices so that test data can be compared across different microprocessors. The original codes have been used several times by other organizations to test a wide range of microcontrollers and microprocessors. After several years of research, LANL is ready to update the benchmark. Changes include: 1. Addition of new codes that allow common software codes to be tested, 2. Addition of new codes that instrument more microprocessor circuitry, 3. Addition of input patterns that allow for a more compressive understanding of how the memory layout affects the sensitivity to radiation-induced faults and better use of automated test pattern generation standards, and 4. Modification of current codes for faster and more resilient detection, reporting and correction of radiation-induced faults. These codes have been tested by LANL researchers over the last few years, which has been published in the open literature. As the code base for the new benchmarks are stable, it is time to release the update to the GitHub repository, where the original codes were released.

Quinn, Heather↗

Uniformly Ordered Binary Decision Algorithm for Benchmark Experiment Correlations in Whisper Validation

When performing a validation exercise for determining the upper subcritical limit of a nuclear criticality safety application, an analyst should select and perform a statistical analysis on a population of benchmark experiments that are neutronically similar to the application. The size of this population should be sufficiently large such that the statistical analysis has a high degree of confidence that the bias plus bias uncertainty (calculational margin) has been accurately quantified. A complication arises because many benchmark experiments share common components, leading to correlations in their measured effective multiplication factors. Correlations between benchmark experiments within the population reduces its predictive power. This motivates the need for methods that consider benchmark experiment correlations and ensure adequate statistical significance of results. The Whisper code is a statistical analysis pack- age that incorporates nuclear data sensitivity coefficients from MCNP to assess benchmark experiment similarity and then performs an extreme-value analysis to estimate the bias plus bias uncertainty. The original methodology in Whisper does not consider the effect of benchmark experiment correlations when making this estimation, and this summary proposes the uniformly ordered binary decision algorithm to address this shortcoming. The original methodology in Whisper computes similarity coefficients ck for an application compared to all benchmark experiments in its library and develops weighting factors for a selected population proportional to the ck values. The methodology can be interpreted as statistically emulating a validation exercise for a particular application where the weighting factors may be viewed as the likelihood that an analyst would include a particular benchmark experiment within the population. The effective sample size of the population is the expected or mean number of benchmark experiments in the population. The uniformly ordered binary decision algorithm identifies clusters of correlated benchmark experiments within the population and then computes adjusted weighting factors based on the magnitude of the correlation coefficients within the cluster to compute a reduced effective sample size accounting for the lower information content because of correlations. Benchmark experiments within the cluster are ordered randomly with equal probability and probabilistic decisions are made as to whether a benchmark. experiment within the cluster should treated as redundant with a previous one; if two redundant benchmark experiments are included, then the conservative worst case bias plus bias uncertainty is used and the pair is counted as a single benchmark experiment in the population. Results are provided for HEU solutions in a research version of the Whisper software using benchmark experiment correlations provided by DICE, the Database for the International Criticality Safety Benchmark Evaluation Project (ICSBEP). These show that there can be a significant increase in the bias plus bias uncertainty because the effective sample size is reduced, and therefore the algorithm, needing to meet sample size requirements, expands the benchmark experiment population by accepting less similar benchmark experiments that would have otherwise not been included.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Methodology and Software for Gross Defect Detection of Spent Nuclear Fuel at the Atucha-I Reactor [Novel Methodology and Software for Spent Fuel Gross Defect Detection at the Atucha-I Reactor]

Here, fuel assemblies in the spent fuel pool are stored by suspending them in two vertically stacked layers at the Atucha Unit 1 nuclear power plant (Atucha-I). This introduces the unique problem of verifying the presence of fuel in either layer without physically moving the fuel assemblies. Given that the facility uses both natural uranium and slightly enriched uranium at 0.85 wt% 235 U and has been in operation since 1974, a wide range of burnups and cooling times can exist in any given pool. A gross defect detection tool, the spent fuel neutron counter (SFNC), has been used at the site to verify the presence of fuel up to burnups of 8000 MWd/t. At higher discharge burnups, the existing signal processing software of the tool was found to fail due to nonlinearity of the source term with burnup.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

A NASA initiative: Software engineering for reliable complex systems

The objective is the development of methods, technology, and skills that will enable NASA to cost-effectively specify, build, and manage reliable software which can evolve and be maintained over an extended period. The need for such software is rooted in the increasing integration of software and computing components into NASA systems. Current NASA Software Engineering expertise was applied toward some of the largest reliable systems including: shuttle launch; ground support; shuttle simulation; minor control; satellite tracking; and scientific data systems. Unfortunately, no theory exists for reliable complex software systems. NASA is seeking to fill this theoretical gap through a number of approaches. One such approach is to conduct research on theoretical foundations for managing complex software systems. It includes: communication models, new and modified paradigms, and life-cycle models. Another approach is research in the theoretical foundations for reliable software development and validation. It focuses upon formal specifications, programming languages, software engineering systems, software reuse, formal verification, and software safety. Further approaches involve benchmarking a NASA software environment, experimentation within the NASA context, evolution of present NASA methodology, and transfer of technology to the space station software support environment.

Holcomb, Lee B.↗