Search NASA⌕ Search

SEARCH · Search NASA

Results for “software accuracy test suite”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Deep in Data: Empirical Data Based Software Accuracy Testing Using the Building America Field Data Repository

This paper describes progress toward developing a usable, standardized, empirical data-based software accuracy test suite using home energy consumption and building description data. Empirical data collected from around the United States have been translated into a uniform Home Performance Extensible Markup Language format that may enable software developers to create translators to their input schemes for efficient access to the data. This could allow for modeling many homes expediently, and thus implementing software accuracy test cases by applying the translated data.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Deep in Data: Empirical Data Based Software Accuracy Testing Using the Building America Field Data Repository: Preprint

An opportunity is available for using home energy consumption and building description data to develop a standardized accuracy test for residential energy analysis tools. That is, to test the ability of uncalibrated simulations to match real utility bills. Empirical data collected from around the United States have been translated into a uniform Home Performance Extensible Markup Language format that may enable software developers to create translators to their input schemes for efficient access to the data. This may facilitate the possibility of modeling many homes expediently, and thus implementing software accuracy test cases by applying the translated data. This paper describes progress toward, and issues related to, developing a usable, standardized, empirical data-based software accuracy test suite.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Twenty Years On!: Updating the IEA BESTEST Building Thermal Fabric Test Cases for ASHRAE Standard 140

ANSI/ASHRAE Standard 140, Standard Method of Test for the Evaluation of Building Energy Analysis Computer Programs applies the IEA BESTEST building thermal fabric test cases and example simulation results originally published in 1995. These software accuracy test cases and their example simulation results, which comprise the first test suite adapted for the initial 2001 version of Standard 140, are approaching their 20th anniversary. In response to the evolution of the state of the art in building thermal fabric modeling since the test cases and example simulation results were developed, work is commencing to update the normative test specification and the informative example results.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Twenty Years On!: Updating the IEA BESTEST Building Thermal Fabric Test Cases for ASHRAE Standard 140: Preprint

ANSI/ASHRAE Standard 140, Standard Method of Test for the Evaluation of Building Energy Analysis Computer Programs applies the IEA BESTEST building thermal fabric test cases and example simulation results originally published in 1995. These software accuracy test cases and their example simulation results, which comprise the first test suite adapted for the initial 2001 version of Standard 140, are approaching their 20th anniversary. In response to the evolution of the state of the art in building thermal fabric modeling since the test cases and example simulation results were developed, work is commencing to update the normative test specification and the informative example results.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

SU-E-T-159: Evaluation of a Patient Specific QA Tool Based On TG119

Purpose: To evaluate the accuracy of a 3D patient specific QA tool by analysis of the results produced from associated software in homogenous phantom and heterogonous patient CT. Methods: IMRT and VMAT plans of five test suites introduced by TG119 were created in ECLIPSE on a solid water phantom. The ten plans -of increasing complexity- were delivered to Delta4 to give a 3D measurement. The Delta4's “Anatomy” software uses the measured dose to back-calculate the energy fluence of the delivered beams, which is used for dose calculation in a patient CT using a pencilbeam algorithm. The effect of the modulated beams' complexity on the accuracy of the “Anatomy” calculation was evaluated. Both measured and Anatomy doses were compared to ECLIPSE calculation using 3% - 3mm gamma criteria.We also tested the effect of heterogeneity by analyzing the results of “Anatomy” calculation on a Brain VMAT and a 3D conformal lung cases. Results: In homogenous phantom, the gamma passing rates were found to be as low as 74.75% for a complex plan with high modulation. The mean passing rates were 91.47% ± 6.35% for “Anatomy” calculation and 99.46% ± 0.62% for Delta4 measurements.As for the heterogeneous cases, the rates were 96.54%±3.67% and 83.87%±9.42% for Brain VMAT and 3D lung respectively. This increased error in the lung case could be due to the use of the pencil beam algorithm as opposed to the AAA used by ECLIPSE.Also, gamma analysis showed high discrepancy along the beam edge in the “Anatomy” calculated results. This suggests a poor beam modeling in the penumbra region. Conclusion: The results show various sources of errors in “Anatomy” calculations. These include beam modeling in the penumbra region, complexity of a modulated beam (shown in homogenous phantom and brain cases) and dose calculation algorithms (3D conformal lung case)

60 APPLIED LIFE SCIENCES↗

Tools for Accurate and Efficient Analysis of Complex Evolutionary Mechanisms in Microbial Genomes (Final Report)

I proposed to develop computationally efficient tools for accurate detection and reconstruction of microbes' complex evolutionary mechanisms, thus enabling rapid and accurate annotation, analysis and understanding of their genomes. To achieve this goal, I proposed to address three aspects. (1) Mathematical modeling. A major challenge facing the accurate detection of HGT is that of distinguishing between these two events on the one hand and other events that have similar "effects." I proposed to develop a novel mathematical approach for distinguishing among these events. Further, I proposed to develop a set of novel optimization criteria for the evolutionary analysis of microbial genomes in the presence of these complex evolutionary events. (2) Algorithm design. In this aspect of the project, I proposed to develop an array of efficient and accurate algorithms for analyzing microbial genomes based on the formulated optimization criteria. Further, I proposed to test the viability of the criteria and the accuracy of the algorithms in an experimental setting using both synthetic as well as biological data. (3) Software development. I proposed the final outcome to be a suite of software tools which implements the mathematical models as well as the algorithms developed.

59 BASIC BIOLOGICAL SCIENCES↗

Automated Comparison of Building Energy Simulation Engines [Slides]

This presentation describes the BEopt comparative test suite, which is a tool that facilitates the automated comparison of building energy simulation engines. It also demonstrates how the test suite is improving the accuracy of building energy simulation programs.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Core Cosmology Library: Precision Cosmological Predictions for LSST

The Core Cosmology Library (CCL) provides routines to compute basic cosmological observables to a high degree of accuracy, which have been verified with an extensive suite of validation tests. Predictions are provided for many cosmological quantities, including distances, angular power spectra, correlation functions, halo bias and the halo mass function through state-of-the-art modeling prescriptions available in the literature. Fiducial specifications for the expected galaxy distributions for the Large Synoptic Survey Telescope (LSST) are also included, together with the capability of computing redshift distributions for a user-defined photometric redshift model. A rigorous validation procedure, based on comparisons between CCL and independent software packages, allows us to establish a well-defined numerical accuracy for each predicted quantity. As a result, predictions for correlation functions of galaxy clustering, galaxy-galaxy lensing and cosmic shear are demonstrated to be within a fraction of the expected statistical uncertainty of the observables for the models and in the range of scales of interest to LSST. CCL is an open source software package written in C, with a python interface and publicly available at this https URL: https://github.com/LSSTDESC/CCL

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

InverseBench: Inverse design benchmark suite that contains inverse problems from science and engineering (InverseBench) v0.0.1

A software package that contains three inverse design blackbox problems to investigate the efficiency and accuracy of inverse design machine learning models. The software contains highly accurate forward machine learning models that can be used to assess the inverse predictions. The package also contains separate test data for each problem. The inverse design problems that are in the package are: airfoil inverse design, scalar boundary reconstruction and photonic surfaces inverse design.

Grbcic, Luka [Lawrence Berkeley National Laborator↗

Testing contamination source identification methods for water distribution networks

In the event of contamination in a water distribution network (WDN), source identification (SI) methods that analyze sensor data can be used to identify the source location(s). Knowledge of the source location and characteristics are important to inform contamination control and cleanup operations. Various SI strategies that have been developed by researchers differ in their underlying assumptions and solution techniques. The following manuscript presents a systematic procedure for testing and evaluating SI methods. The performance of these SI methods is affected by various factors including the size of WDN model, measurement error, modeling error, time and number of contaminant injections, and time and number of measurements. This paper includes test cases that vary these factors and evaluates three SI methods on the basis of accuracy and specificity. The tests are used to review and compare these different SI methods, highlighting their strengths in handling various identification scenarios. These SI methods and a testing framework that includes the test cases and analysis tools presented in this paper have been integrated into EPA’s Water Security Toolkit (WST), a suite of software tools to help researchers and others in the water industry evaluate and plan various response strategies in case of a contamination incident. Lastly, a set of recommendations are made for users to consider when working with different categories of SI methods.

54 ENVIRONMENTAL SCIENCES↗

GradDFT. A software library for machine learning enhanced density functional theory

Density functional theory (DFT) stands as a cornerstone method in computational quantum chemistry and materials science due to its remarkable versatility and scalability. Yet, it suffers from limitations in accuracy, particularly when dealing with strongly correlated systems. To address these shortcomings, recent work has begun to explore how machine learning can expand the capabilities of DFT: an endeavor with many open questions and technical challenges. In this work, we present GradDFT a fully differentiable JAX-based DFT library, enabling quick prototyping and experimentation with machine learning-enhanced exchange–correlation energy functionals. GradDFT employs a pioneering parametrization of exchange–correlation functionals constructed using a weighted sum of energy densities, where the weights are determined using neural networks. Moreover, GradDFT encompasses a comprehensive suite of auxiliary functions, notably featuring a just-in-time compilable and fully differentiable self-consistent iterative procedure. To support training and benchmarking efforts, we additionally compile a curated dataset of experimental dissociation energies of dimers, half of which contain transition metal atoms characterized by strong electronic correlations. The software library is tested against experimental results to study the generalization capabilities of a neural functional across potential energy surfaces and atomic species, as well as the effect of training data noise on the resulting model accuracy.

Chemistry↗

RELAP5-3D Resolution of Known Restart/Backup Issues

The state-of-the-art nuclear reactor system safety analysis computer program developed at the Idaho National Laboratory (INL), RELAP5-3D, continues to adapt to changes in computer hardware and software and to develop to meet the ever-expanding needs of the nuclear industry. To continue at the forefront, code testing must evolve with both code and industry developments, and it must work correctly. To best ensure this, the processes of Software Verification and Validation (V&V) are applied. Verification compares coding against its documented algorithms and equations and compares its calculations against analytical solutions and the method of manufactured solutions. A form of this, sequential verification, checks code specifications against coding only when originally written then applies regression testing which compares code calculations between consecutive updates or versions on a set of test cases to check that the performance does not change. A sequential verification testing system was specially constructed for RELAP5-3D to both detect errors with extreme accuracy and cover all nuclear-plant-relevant code features. Detection is provided through a “verification file” that records double precision sums of key variables. Coverage is provided by a test suite of input decks that exercise code features and capabilities necessary to model a nuclear power plant. A matrix of test features and short-running cases that exercise them is presented. This testing system is used to test base cases (called null testing) as well as restart and backup cases. It can test RELAP5-3D performance in both standalone and coupled (through PVM to other codes) runs. Application of verification testing revealed numerous restart and backup issues in both standalone and couple modes. This document reports the resolution of these issues.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Enhanced Sequential Search Methodology for Identifying Cost-Optimal Building Pathways

The BEopt software is a building energy optimization tool that generates a cost-optimal path of building designs from a reference building up to zero-net energy. It employs a sequential search methodology to account for complex energy interactions between building efficiency measures. Enhancement strategies to this search methodology are developed to increase accuracy (ability to identify the true cost-optimal curve) and speed (number of required energy simulations). A test suite of optimizations is used to gauge the effectiveness of each strategy. Combinations of strategies are assembled into packages, ranging from conservative to aggressive, with so up to 71% fewer required simulations are required.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Sensitive and Specific Peak Detection for SELDI-TOF Mass Spectrometry Using a Wavelet/Neural-Network Based Approach

SELDI-TOF mass spectrometer’s compact size and automated, high throughput design have been attractive to clinical researchers, and the platform has seen steady-use in biomarker studies. Despite new algorithms and preprocessing pipelines that have been developed to address reproducibility issues, visual inspection of the results of SELDI spectra preprocessing by the best algorithms still shows miscalled peaks and systematic sources of error. This suggests that there continues to be problems with SELDI preprocessing. In this work, we study the preprocessing of SELDI in detail and introduce improvements. While many algorithms, including the vendor supplied software, can identify peak clusters of specific mass (or m/z) in groups of spectra with high specificity and low false discover rate (FDR), the algorithms tend to underperform estimating the exact prevalence and intensity of peaks in those clusters. Thus group differences that at first appear very strong are shown, after careful and laborious hand inspection of the spectra, to be less than significant. Here we introduce a wavelet/neural network based algorithm which mimics what a team of expert, human users would call for peaks in each of several hundred spectra in a typical SELDI clinical study. The wavelet denoising part of the algorithm optimally smoothes the signal in each spectrum according to an improved suite of signal processing algorithms previously reported (the LibSELDI toolbox under development). The neural network part of the algorithm combines those results with the raw signal and a training dataset of expertly called peaks, to call peaks in a test set of spectra with approximately 95% accuracy. The new method was applied to data collected from a study of cervical mucus for the early detection of cervical cancer in HPV infected women. The method shows promise in addressing the ongoing SELDI reproducibility issues.

59 BASIC BIOLOGICAL SCIENCES↗

Wrapper for the optimization of cross-section generation in OpenMC

The "Wrapper for the optimization of cross-section generation in OpenMC" is Python-based software that runs the open-source Monte Carlo code OpenMC (https://docs.openmc.org/en/stable/ ) to generate cross-sections for any reactor geometry. The wrapper then optimizes those cross sections. The optimization aims to choose an energy group structure and a scattering representation that maximize accuracy with respect to continuous-energy results while avoiding significant computational expense. It then outputs these cross-sections in an ISOXML format readable by the Idaho National Lab code suite MOOSE (Olin William Calvin, Mark D DeHart, “Architecture for the Performance of Nuclear Fuel Depletion Calculations”, Idaho National Laboratory report, November 2019). The expected use-cases of this software include: -finding the best group structure and scattering representation for a specific reactor -testing the appropriateness of energy group structures for different reactor types -comparing energy group structures and scattering representations to each other -generating cross sections for use in deterministic codes, including ones found in the MOOSE suite The example reactor geometry included in this release is a generic reactor design, not based on any reactor in existence or in development. It was fabricated for the sole purpose of being a “testbed-geometry” upon which to develop this tool. Since the tool is designed to be generic, the nature of the test geometry is not very important, however, it is valuable to include as an example for users who are unfamiliar with developing reactor geometries for OpenMC.

Kreher, Miriam↗

Effective and efficient optics inspection approach using machine learning algorithms

The Final Optics Damage Inspection (FODI) system automatically acquires and utilizes the Optics Inspection (OI) system to analyze images of the final optics at the National Ignition Facility (NIF). During each inspection cycle up to 1000 images acquired by FODI are examined by OI to identify and track damage sites on the optics. The process of tracking growing damage sites on the surface of an optic can be made more effective by identifying and removing signals associated with debris or reflections. The manual process to filter these false sites is daunting and time consuming. In this paper we discuss the use of machine learning tools and data mining techniques to help with this task. We describe the process to prepare a data set that can be used for training and identifying hardware reflections in the image data. In order to collect training data, the images are first automatically acquired and analyzed with existing software and then relevant features such as spatial, physical and luminosity measures are extracted for each site. A subset of these sites is 'truthed' or manually assigned a class to create training data. A supervised classification algorithm is used to test if the features can predict the class membership of new sites. A suite of self-configuring machine learning tools called 'Avatar Tools' is applied to classify all sites. To verify, we used 10-fold cross correlation and found the accuracy was above 99%. This substantially reduces the number of false alarms that would otherwise be sent for more extensive investigation.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

CTF Theory Manual: Version 4.4

Coolant-Boiling in Rod Arrays – Two Fluids (COBRA-TF) is a thermal/hydraulic (T/H) simulation code designed for light-water reactor (LWR) vessel analysis. It uses a two-fluid, three-field (i.e., fluid film, fluid drops, and vapor) modeling approach. Both subchannel and 3D Cartesian forms of its governing equations are available for the solution. The code was originally developed by Pacific Northwest Laboratory in 1980 and has been used and modified by several institutions over the last few decades. COBRA-TF also found use at the Pennsylvania State University (PSU) by the Reactor Dynamics and Fuel Modeling Group (RDFMG) and has been improved, updated, and subsequently rebranded as CTF. CTF was later adopted in the early 2010s by Oak Ridge National Laboratory (ORNL) for use in the Consortium for Advanced Simulation of Light Water Reactors (CASL) program, which led to a significant advancement of the code’s software quality, modeling accuracy, testing systems, and capabilities for improved support in modeling common LWR nominal and transient behavior. As part of the improvement process, it was necessary to generate sufficient documentation for the public domain code which had lacked such material upon being adopted by RDFMG. This document serves as a theory manual for CTF, detailing the many two-phase heat transfer, drag, and important accident scenario models contained in the code, as well as the numerical solution process utilized. Additional documents available in the CTF documentation suite include the user manual and the verification and validation manual.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗