Search NASA⌕ Search

SEARCH · Search NASA

Results for “software accuracy test suite”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Deep in Data: Empirical Data Based Software Accuracy Testing Using the Building America Field Data Repository

This paper describes progress toward developing a usable, standardized, empirical data-based software accuracy test suite using home energy consumption and building description data. Empirical data collected from around the United States have been translated into a uniform Home Performance Extensible Markup Language format that may enable software developers to create translators to their input schemes for efficient access to the data. This could allow for modeling many homes expediently, and thus implementing software accuracy test cases by applying the translated data.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Deep in Data: Empirical Data Based Software Accuracy Testing Using the Building America Field Data Repository: Preprint

An opportunity is available for using home energy consumption and building description data to develop a standardized accuracy test for residential energy analysis tools. That is, to test the ability of uncalibrated simulations to match real utility bills. Empirical data collected from around the United States have been translated into a uniform Home Performance Extensible Markup Language format that may enable software developers to create translators to their input schemes for efficient access to the data. This may facilitate the possibility of modeling many homes expediently, and thus implementing software accuracy test cases by applying the translated data. This paper describes progress toward, and issues related to, developing a usable, standardized, empirical data-based software accuracy test suite.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Twenty Years On!: Updating the IEA BESTEST Building Thermal Fabric Test Cases for ASHRAE Standard 140

ANSI/ASHRAE Standard 140, Standard Method of Test for the Evaluation of Building Energy Analysis Computer Programs applies the IEA BESTEST building thermal fabric test cases and example simulation results originally published in 1995. These software accuracy test cases and their example simulation results, which comprise the first test suite adapted for the initial 2001 version of Standard 140, are approaching their 20th anniversary. In response to the evolution of the state of the art in building thermal fabric modeling since the test cases and example simulation results were developed, work is commencing to update the normative test specification and the informative example results.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Twenty Years On!: Updating the IEA BESTEST Building Thermal Fabric Test Cases for ASHRAE Standard 140: Preprint

ANSI/ASHRAE Standard 140, Standard Method of Test for the Evaluation of Building Energy Analysis Computer Programs applies the IEA BESTEST building thermal fabric test cases and example simulation results originally published in 1995. These software accuracy test cases and their example simulation results, which comprise the first test suite adapted for the initial 2001 version of Standard 140, are approaching their 20th anniversary. In response to the evolution of the state of the art in building thermal fabric modeling since the test cases and example simulation results were developed, work is commencing to update the normative test specification and the informative example results.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

SU-E-T-159: Evaluation of a Patient Specific QA Tool Based On TG119

Purpose: To evaluate the accuracy of a 3D patient specific QA tool by analysis of the results produced from associated software in homogenous phantom and heterogonous patient CT. Methods: IMRT and VMAT plans of five test suites introduced by TG119 were created in ECLIPSE on a solid water phantom. The ten plans -of increasing complexity- were delivered to Delta4 to give a 3D measurement. The Delta4's “Anatomy” software uses the measured dose to back-calculate the energy fluence of the delivered beams, which is used for dose calculation in a patient CT using a pencilbeam algorithm. The effect of the modulated beams' complexity on the accuracy of the “Anatomy” calculation was evaluated. Both measured and Anatomy doses were compared to ECLIPSE calculation using 3% - 3mm gamma criteria.We also tested the effect of heterogeneity by analyzing the results of “Anatomy” calculation on a Brain VMAT and a 3D conformal lung cases. Results: In homogenous phantom, the gamma passing rates were found to be as low as 74.75% for a complex plan with high modulation. The mean passing rates were 91.47% ± 6.35% for “Anatomy” calculation and 99.46% ± 0.62% for Delta4 measurements.As for the heterogeneous cases, the rates were 96.54%±3.67% and 83.87%±9.42% for Brain VMAT and 3D lung respectively. This increased error in the lung case could be due to the use of the pencil beam algorithm as opposed to the AAA used by ECLIPSE.Also, gamma analysis showed high discrepancy along the beam edge in the “Anatomy” calculated results. This suggests a poor beam modeling in the penumbra region. Conclusion: The results show various sources of errors in “Anatomy” calculations. These include beam modeling in the penumbra region, complexity of a modulated beam (shown in homogenous phantom and brain cases) and dose calculation algorithms (3D conformal lung case)

60 APPLIED LIFE SCIENCES↗

Tools for Accurate and Efficient Analysis of Complex Evolutionary Mechanisms in Microbial Genomes (Final Report)

I proposed to develop computationally efficient tools for accurate detection and reconstruction of microbes' complex evolutionary mechanisms, thus enabling rapid and accurate annotation, analysis and understanding of their genomes. To achieve this goal, I proposed to address three aspects. (1) Mathematical modeling. A major challenge facing the accurate detection of HGT is that of distinguishing between these two events on the one hand and other events that have similar "effects." I proposed to develop a novel mathematical approach for distinguishing among these events. Further, I proposed to develop a set of novel optimization criteria for the evolutionary analysis of microbial genomes in the presence of these complex evolutionary events. (2) Algorithm design. In this aspect of the project, I proposed to develop an array of efficient and accurate algorithms for analyzing microbial genomes based on the formulated optimization criteria. Further, I proposed to test the viability of the criteria and the accuracy of the algorithms in an experimental setting using both synthetic as well as biological data. (3) Software development. I proposed the final outcome to be a suite of software tools which implements the mathematical models as well as the algorithms developed.

59 BASIC BIOLOGICAL SCIENCES↗

Automated Comparison of Building Energy Simulation Engines [Slides]

This presentation describes the BEopt comparative test suite, which is a tool that facilitates the automated comparison of building energy simulation engines. It also demonstrates how the test suite is improving the accuracy of building energy simulation programs.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Core Cosmology Library: Precision Cosmological Predictions for LSST

The Core Cosmology Library (CCL) provides routines to compute basic cosmological observables to a high degree of accuracy, which have been verified with an extensive suite of validation tests. Predictions are provided for many cosmological quantities, including distances, angular power spectra, correlation functions, halo bias and the halo mass function through state-of-the-art modeling prescriptions available in the literature. Fiducial specifications for the expected galaxy distributions for the Large Synoptic Survey Telescope (LSST) are also included, together with the capability of computing redshift distributions for a user-defined photometric redshift model. A rigorous validation procedure, based on comparisons between CCL and independent software packages, allows us to establish a well-defined numerical accuracy for each predicted quantity. As a result, predictions for correlation functions of galaxy clustering, galaxy-galaxy lensing and cosmic shear are demonstrated to be within a fraction of the expected statistical uncertainty of the observables for the models and in the range of scales of interest to LSST. CCL is an open source software package written in C, with a python interface and publicly available at this https URL: https://github.com/LSSTDESC/CCL

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

InverseBench: Inverse design benchmark suite that contains inverse problems from science and engineering (InverseBench) v0.0.1

A software package that contains three inverse design blackbox problems to investigate the efficiency and accuracy of inverse design machine learning models. The software contains highly accurate forward machine learning models that can be used to assess the inverse predictions. The package also contains separate test data for each problem. The inverse design problems that are in the package are: airfoil inverse design, scalar boundary reconstruction and photonic surfaces inverse design.

Grbcic, Luka [Lawrence Berkeley National Laborator↗

Testing contamination source identification methods for water distribution networks

In the event of contamination in a water distribution network (WDN), source identification (SI) methods that analyze sensor data can be used to identify the source location(s). Knowledge of the source location and characteristics are important to inform contamination control and cleanup operations. Various SI strategies that have been developed by researchers differ in their underlying assumptions and solution techniques. The following manuscript presents a systematic procedure for testing and evaluating SI methods. The performance of these SI methods is affected by various factors including the size of WDN model, measurement error, modeling error, time and number of contaminant injections, and time and number of measurements. This paper includes test cases that vary these factors and evaluates three SI methods on the basis of accuracy and specificity. The tests are used to review and compare these different SI methods, highlighting their strengths in handling various identification scenarios. These SI methods and a testing framework that includes the test cases and analysis tools presented in this paper have been integrated into EPA’s Water Security Toolkit (WST), a suite of software tools to help researchers and others in the water industry evaluate and plan various response strategies in case of a contamination incident. Lastly, a set of recommendations are made for users to consider when working with different categories of SI methods.

54 ENVIRONMENTAL SCIENCES↗

GradDFT. A software library for machine learning enhanced density functional theory

Density functional theory (DFT) stands as a cornerstone method in computational quantum chemistry and materials science due to its remarkable versatility and scalability. Yet, it suffers from limitations in accuracy, particularly when dealing with strongly correlated systems. To address these shortcomings, recent work has begun to explore how machine learning can expand the capabilities of DFT: an endeavor with many open questions and technical challenges. In this work, we present GradDFT a fully differentiable JAX-based DFT library, enabling quick prototyping and experimentation with machine learning-enhanced exchange–correlation energy functionals. GradDFT employs a pioneering parametrization of exchange–correlation functionals constructed using a weighted sum of energy densities, where the weights are determined using neural networks. Moreover, GradDFT encompasses a comprehensive suite of auxiliary functions, notably featuring a just-in-time compilable and fully differentiable self-consistent iterative procedure. To support training and benchmarking efforts, we additionally compile a curated dataset of experimental dissociation energies of dimers, half of which contain transition metal atoms characterized by strong electronic correlations. The software library is tested against experimental results to study the generalization capabilities of a neural functional across potential energy surfaces and atomic species, as well as the effect of training data noise on the resulting model accuracy.

Chemistry↗

Toward an IMU-based Space Suit Motion Capture System

Spacesuits are complex engineering systems that sustain human health and enable performance outside of Earth-like environments. These systems must support human mobility and physical workload demands while minimizing injury risk during extravehicular activity (EVA). Future EVA on the lunar surface during the Artemis program is expected to be more frequent and require higher physical workloads than previous EVAs during the ISS, Shuttle, or Apollo programs. Hence it is important to optimize future as well as current spacesuits to be efficient and comfortable for the success of space and planetary missions. To enable this, an efficient method is needed to test these spacesuits on the ground.When testing spacesuits in ground environments, it is often necessary to understand the kinematics of the suit to validate the design against relevant requirements or characterize the physical workload necessary to operate the suit. This is a challenging task for traditional optical motion capture (OMC) approaches: suit-mounted OMC markers are easily occluded by the subject or environment and may become detached during testing. Controlling lighting and reflectivity of objects in the motion capture volume is also difficult. Fixed-position OMC cameras also constrain testing to a small and contrived laboratory environment, disallowing kinematics capture in field environments.One promising alternative is the use of suit-mounted inertial measurement units (IMUs). These sensors are small, unobtrusive, and portable, but come at the cost of increased sensor noise and complexity of the software and mathematics to analyze the collected data. To this end, engineers at NASA are developing the Augmented Suit Inverse Kinematics (ASIK) system, a complete motion capture methodand inverse kinematics solver which relies solely on a network of wireless IMUs attached to the major kinematic segments of the spacesuit. The ASIK modeling language allows for the simple inclusion of probabilistic priors such as suit size and shape or IMU positions and rotations. Furthermore, to increase accuracy and reduce operational overhead to use this motion capture approach, the developed inverse kinematics solver exploits so-called self-calibratingalgorithmic techniques, which reduce the need for precise alignment of the sensors on the segments or scripted functional calibration procedures. The ASIK system was tested in a 7-subject pilot study. Each subject donned NASA’s new prototype exploration spacesuit in the Active Response Gravity Offload System (ARGOS) facility at the NASA Johnson Space Center. The subjects were outfitted with a set of 14 APDM (Portland, OR, USA) Opal IMUs, 12 of which were used in the ASIK model to estimate lower body and trunk kinematics. The subjects were also outfitted with a set of reflective OMC markers and traditional OMC data was collected and processed. Presented results will include characterization of ASIK-derived suit joint angles accuracy against an optical motion capture datum. Discussion of these results, as well as discussion of system calibration and nuances of mathematical observability, will be included.If successful, IMU-based motion capture will enable testing and validation of spacesuits more frequently, with less overhead, in more extreme environments. Future work will apply these techniques to common spacesuit testing tasks, such as gait, mobility, and balance assessment, physical workload characterization, and ergonomics evaluations.

Timothy Mcgrath↗

Validation of the 5th and 95th Percentile Hybrid III Anthropomorphic Test Device Finite Element Model

NASA spacecraft design requirements for occupant protection are a combination of the Brinkley criteria and injury metrics extracted from anthropomorphic test devices (ATD's). For the ATD injury metrics, the requirements specify the use of the 5th percentile female Hybrid III and the 95th percentile male Hybrid III. Furthermore, each of these ATD's is required to be fitted with an articulating pelvis and a straight spine. The articulating pelvis is necessary for the ATD to fit into spacecraft seats, while the straight spine is required as injury metrics for vertical accelerations are better defined for this configuration. The requirements require that physical testing be performed with both ATD's to demonstrate compliance. Before compliance testing can be conducted, extensive modeling and simulation are required to determine appropriate test conditions, simulate conditions not feasible for testing, and assess design features to better ensure compliance testing is successful. While finite element (FE) models are currently available for many of the physical ATD's, currently there are no complete models for either the 5th percentile female or the 95th percentile male Hybrid III with a straight spine and articulating pelvis. The purpose of this work is to assess the accuracy of the existing Livermore Software Technology Corporation's FE models of the 5th and 95th percentile ATD's. To perform this assessment, a series of tests will be performed at Wright Patterson Air Force Research Lab using their horizontal impact accelerator sled test facility. The ATD's will be placed in the Orion seat with a modified-advanced-crew-escape-system (MACES) pressure suit and helmet, and driven with loadings similar to what is expected for the actual Orion vehicle during landing, launch abort, and chute deployment. Test data will be compared to analytical predictions and modelling uncertainty factors will be determined for each injury metric. Additionally, the test data will be used to further improve the FE model, particularly in the areas of the ATD neck components, harness, and suit and helmet effects.

Lawrence, C.↗

RELAP5-3D Resolution of Known Restart/Backup Issues

The state-of-the-art nuclear reactor system safety analysis computer program developed at the Idaho National Laboratory (INL), RELAP5-3D, continues to adapt to changes in computer hardware and software and to develop to meet the ever-expanding needs of the nuclear industry. To continue at the forefront, code testing must evolve with both code and industry developments, and it must work correctly. To best ensure this, the processes of Software Verification and Validation (V&V) are applied. Verification compares coding against its documented algorithms and equations and compares its calculations against analytical solutions and the method of manufactured solutions. A form of this, sequential verification, checks code specifications against coding only when originally written then applies regression testing which compares code calculations between consecutive updates or versions on a set of test cases to check that the performance does not change. A sequential verification testing system was specially constructed for RELAP5-3D to both detect errors with extreme accuracy and cover all nuclear-plant-relevant code features. Detection is provided through a “verification file” that records double precision sums of key variables. Coverage is provided by a test suite of input decks that exercise code features and capabilities necessary to model a nuclear power plant. A matrix of test features and short-running cases that exercise them is presented. This testing system is used to test base cases (called null testing) as well as restart and backup cases. It can test RELAP5-3D performance in both standalone and coupled (through PVM to other codes) runs. Application of verification testing revealed numerous restart and backup issues in both standalone and couple modes. This document reports the resolution of these issues.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Enhanced Sequential Search Methodology for Identifying Cost-Optimal Building Pathways

The BEopt software is a building energy optimization tool that generates a cost-optimal path of building designs from a reference building up to zero-net energy. It employs a sequential search methodology to account for complex energy interactions between building efficiency measures. Enhancement strategies to this search methodology are developed to increase accuracy (ability to identify the true cost-optimal curve) and speed (number of required energy simulations). A test suite of optimizations is used to gauge the effectiveness of each strategy. Combinations of strategies are assembled into packages, ranging from conservative to aggressive, with so up to 71% fewer required simulations are required.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Sensitive and Specific Peak Detection for SELDI-TOF Mass Spectrometry Using a Wavelet/Neural-Network Based Approach

SELDI-TOF mass spectrometer’s compact size and automated, high throughput design have been attractive to clinical researchers, and the platform has seen steady-use in biomarker studies. Despite new algorithms and preprocessing pipelines that have been developed to address reproducibility issues, visual inspection of the results of SELDI spectra preprocessing by the best algorithms still shows miscalled peaks and systematic sources of error. This suggests that there continues to be problems with SELDI preprocessing. In this work, we study the preprocessing of SELDI in detail and introduce improvements. While many algorithms, including the vendor supplied software, can identify peak clusters of specific mass (or m/z) in groups of spectra with high specificity and low false discover rate (FDR), the algorithms tend to underperform estimating the exact prevalence and intensity of peaks in those clusters. Thus group differences that at first appear very strong are shown, after careful and laborious hand inspection of the spectra, to be less than significant. Here we introduce a wavelet/neural network based algorithm which mimics what a team of expert, human users would call for peaks in each of several hundred spectra in a typical SELDI clinical study. The wavelet denoising part of the algorithm optimally smoothes the signal in each spectrum according to an improved suite of signal processing algorithms previously reported (the LibSELDI toolbox under development). The neural network part of the algorithm combines those results with the raw signal and a training dataset of expertly called peaks, to call peaks in a test set of spectra with approximately 95% accuracy. The new method was applied to data collected from a study of cervical mucus for the early detection of cervical cancer in HPV infected women. The method shows promise in addressing the ongoing SELDI reproducibility issues.

59 BASIC BIOLOGICAL SCIENCES↗

Wrapper for the optimization of cross-section generation in OpenMC

The "Wrapper for the optimization of cross-section generation in OpenMC" is Python-based software that runs the open-source Monte Carlo code OpenMC (https://docs.openmc.org/en/stable/ ) to generate cross-sections for any reactor geometry. The wrapper then optimizes those cross sections. The optimization aims to choose an energy group structure and a scattering representation that maximize accuracy with respect to continuous-energy results while avoiding significant computational expense. It then outputs these cross-sections in an ISOXML format readable by the Idaho National Lab code suite MOOSE (Olin William Calvin, Mark D DeHart, “Architecture for the Performance of Nuclear Fuel Depletion Calculations”, Idaho National Laboratory report, November 2019). The expected use-cases of this software include: -finding the best group structure and scattering representation for a specific reactor -testing the appropriateness of energy group structures for different reactor types -comparing energy group structures and scattering representations to each other -generating cross sections for use in deterministic codes, including ones found in the MOOSE suite The example reactor geometry included in this release is a generic reactor design, not based on any reactor in existence or in development. It was fabricated for the sole purpose of being a “testbed-geometry” upon which to develop this tool. Since the tool is designed to be generic, the nature of the test geometry is not very important, however, it is valuable to include as an example for users who are unfamiliar with developing reactor geometries for OpenMC.

Kreher, Miriam↗