Search NASA⌕ Search

SEARCH · Search NASA

Results for “regression testing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Application-level regression testing framework using Jenkins

Monitoring and testing for regression of large-scale systems such as the NCSA's Blue Waters supercomputer are challenging tasks. In this paper, we describe the solution we came up with to perform those tasks. The goal was to find an automated solution for running user-level regression tests to evaluate system usability and performance. Jenkins, an automation server software, was chosen for its versatility, large user base, and multitude of plugins including collecting data and plotting test results over time. We also describe our Jenkins deployment to launch and monitor jobs on remote HPC system, perform authentication with one-time password, and integrate with our LDAP server for its authorization. We show some use cases and describe our best practices for successfully using Jenkins as a user-level system-wide regression testing and monitoring framework for large supercomputer systems.

97 MATHEMATICS AND COMPUTING↗

RAVEN regression tests' description

Regression tests for the Python RAVEN framework are found in raven/tests/framework. There is a hierarchy of folders with tests collected by similar testing. Every test is described in a special XML node (< TestInfo >) within the < Simulation > block.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DYNA3D/ParaDyn Regression Test Suite Inventory

The following table constitutes an initial assessment of feature coverage across the regression test suite used for DYNA3D and ParaDyn. It documents the regression test suite at the time of preliminary release 16.1 in September 2016. The columns of the table represent groupings of functionalities, e.g., material models. Each problem in the test suite is represented by a row in the table. All features exercised by the problem are denoted by a check mark (√) in the corresponding column. The definition of “feature” has not been subdivided to its smallest unit of user input, e.g., algorithmic parameters specific to a particular type of contact surface. This represents a judgment to provide code developers and users a reasonable impression of feature coverage without expanding the width of the table by several multiples. All regression testing is run in parallel, typically with eight processors, except problems involving features only available in serial mode. Many are strictly regression tests acting as a check that the codes continue to produce adequately repeatable results as development unfolds; compilers change and platforms are replaced. A subset of the tests represents true verification problems that have been checked against analytical or other benchmark solutions. Users are welcomed to submit documented problems for inclusion in the test suite, especially if they are heavily exercising, and dependent upon, features that are currently underrepresented.

97 MATHEMATICS AND COMPUTING↗

Verification and Regression Testing of a Physically Stabilized Layered Solid Element Formulation in DYNA3D/Paradyn

Recent modeling of filament-wound composite structures drove the need for a new element type within the finite element code DYNA3D/Paradyn. This new layered solid element was implemented and designed to capture the kinematics and constitutive be havior of various lamina layers defined with arbitrary orientations and volume fractions within the element to accurately model a laminated composite material. The new el ement uses a single integration point in each of the lamina layers defined to capture the constitutive behavior, and a novel physical stabilization routine is used to prevent hourglassing while mitigating shear and volumetric locking. This report documents the verification testing and the regression tests added to the DYNA3D/Paradyn software quality assurance test suite to assess the proper implementation of the new layered solid element. There were a total of 27 tests added the DYNA3D test suite, which consist of problems using the layered solid element with a single layer or multiple layers for isotropic and orthotropic material models. These tests are a combination of simple kinematically driven patch tests, beam bending tests, plate bending tests, and more complicated problems. Ultimately, the testing done verifies that the element behaves as expected and is implemented correctly.

42 ENGINEERING↗

Summary of Documentation for DYNA3D-ParaDyn's Software Quality Assurance Regression Test Problems

The Software Quality Assurance (SQA) regression test suite for DYNA3D (Zywicz and Lin, 2015) and ParaDyn (DeGroot, et al., 2015) currently contains approximately 600 problems divided into 21 suites, and is a required component of ParaDyn’s SQA plan (Ferencz and Oliver, 2013). The regression suite allows developers to ensure that software modifications do not unintentionally alter the code response. The entire regression suite is run prior to permanently incorporating any software modification or addition. When code modifications alter test problem results, the specific cause must be determined and fully understood before the software changes and revised test answers can be incorporated. The regression suite is executed on LLNL platforms using a Python script and an associated data file. The user specifies the DYNA3D or ParaDyn executable, number of processors to use, test problems to run, and other options to the script. The data file details how each problem and its answer extraction scripts are executed. For each problem in the regression suite there exists an input deck, an eight-processor partition file, an answer file, and various extraction scripts. These scripts assemble a temporary answer file in a specific format from the simulation results. The temporary and stored answer files are compared to a specific level of numerical precision, and when differences are detected the test problem is flagged as failed. Presently, numerical results are stored and compared to 16 digits. At this accuracy level different processor types, compilers, number of partitions, etc. impact the results to various degrees. Thus, for consistency purposes the regression suite is run with ParaDyn using 8 processors on machines with a specific processor type (currently the Intel Xeon E5530 processor). For non-parallel regression problems, i.e., the two XFEM problems, DYNA3D is used instead. When environments or platforms change, executables using the current source code and the new resource are created and the regression suite is run. If differences in answers arise, the new answers are retained provided that the differences are inconsequential. This bootstrap approach allows the test suite answers to evolve in a controlled manner with a high level of confidence. Developers also run the entire regression suite with (serial) DYNA3D. While these results normally differ from the stored (parallel) answers, abnormal termination or wildly different values are strong indicators of potential issues.

42 ENGINEERING↗

FORCE Regression Testing

Via programs including the Light Water Reactor Sustainability and Integrated Energy Systems, the U.S. Department of Energy has invested in the Framework for Optimization of ResourCes and Economics (FORCE) software framework (Idaho National Laboratory 2024a) for the technical and economic analysis of nuclear-integrated energy systems (IES). Nuclear IES expand the use of nuclear from traditional baseload electricity generation to a flexible and adaptive source of combined heat and power. Nuclear heat can be used in the production of a variety of energy currencies such as hydrogen and ammonia as well as other heat applications including water desalination and district heating. FORCE is designed with the intent to provide interconnected analysis tools that enable the accurate technical and economic assessment of specific nuclear IES configurations for individual energy markets. FORCE consists of three main analysis pathways: HYBRID (Idaho National Laboratory 2024b), which contains high-resolution physical models for IES; Holistic Energy Resource Optimization Network (HERON) (Idaho National Laboratory 2024c), which analyzes IES long-term economic viability; and Optimization of Real-time Capacity Allocation (ORCA) (Idaho National Laboratory 2024d), designed for real-time control of IES via digital twins and optimal decision making, including autonomous and remote operation research. Development of the FORCE ecosystem is guided by three pillars: capability, which assures that the computational requirements of IES analysis are met by the software tools; reliability, which provides for consistent code performance and expected behaviors; and accessibility, which lowers the barrier to entry for using the software and accelerates analysis by users beyond the FORCE primary developers. Reliability of the FORCE ecosystem is established according to the American Nuclear Society?s Nuclear Quality Assurance (NQA-1) program [American Society of Mechanical Engineers 1982], with specific levels of software quality assurance (SQA) within NQA-1 applied to each software tool in FORCE. As the tools within FORCE have matured, some integration algorithms to accurately connect the software tools for holistic analysis have been developed and deployed within the FORCE software repository. In accordance with NQA-1 standards, regression tests are required to guarantee the software performs consistently even when new capabilities are added to the software. In this report, we document the deployment of both unit tests, which test the consistent behavior of small pieces of the FORCE code base, as well as integration tests, which test the consistent performance of full use cases for the FORCE integration algorithms. We further document the encapsulation of these tests within a test harness, which collectively checks for each successful test completion on demand. Finally, we document the automation of the test harness using GitHub Actions [GitHub 2024], which require all tests succeed before any new capability or other changes can be added to the FORCE integration software

97 MATHEMATICS AND COMPUTING↗

Verification Testing of Body Forces due to a Prescribed Angular Velocity in DYNA3D/Paradyn

This report documents the verification testing and regression testing done on a new feature in DYNA3D/Paradyn, which allows users to prescribe body force loads based upon an angular velocity. This feature is unique in that the direction of the angular velocity vector follows the unit vector formed by two coordinate points associated with two nodes or the average coordinates of two separate small collection of nodes. The angular velocity direction will follow the directional vector defined by these nodes while the angular velocity magnitude is defined by a load curve. A simple single element verification test was performed to determine the correct implementation of this feature, and two separate regression tests were added to the DYNA3D Software Quality Assur ance test suite. The nodal positions, velocities, and accelerations from the solution of the single element verification test compare well to analytically derived values of those nodal quantities. The regression tests serve as good examples of this new feature’s use case and were consequently added to the SQA test suite to ensure that further modifications of the DYNA3D source code do not unintentionally change the generated baseline answers.

42 ENGINEERING↗

Code Verification and Solution Verification framework in pin-resolved neutron transport code MPACT

Program verification in scientific computing encompasses the application of formal and mathematical techniques to a scientific computing code for its credibility, accuracy, and validity. Code Verification identifies bugs and performance issues in the software development stage. Solution Verification assesses the applicability of the code and the accuracy of the solution to problems of interest. Both activities utilize application cases and quantify the error against prescribed acceptance criteria. However, simply executing more application cases does not guarantee stronger or more comprehensive credibility. Here, we establish a verification framework that involves Code Verification and Solution Verification, both of which work together such that the overarching goal of “converge to the correct answer for the intended application” can be reasonably inferred. The application of such a verification framework is demonstrated using the pin-resolved neutron transport code MPACT, where standard unit tests and regression tests are covered, and where the Method of Exact Solutions and the Method of Manufactured Solutions are successfully used. Additionally, the applicability of Method of Manufactured Solutions is extended to the OECD/NEA C5G7 benchmark problems of practical material and geometric configurations. Solution Verification activities are demonstrated on a practical hierarchy of application models of increasing complexity ranging from 2D pin cell problems to 3D assembly problems. The convergence behavior and rate of convergence with respect to each individual variable are studied and provided. This framework can be adapted broadly to other fields involving scientific computing codes.

97 MATHEMATICS AND COMPUTING↗

Results of a Geant4 benchmarking study for bio‐medical applications, performed with the G4‐Med system

Geant4, a Monte Carlo Simulation Toolkit extensively used in bio-medical physics, is in continuous evolution to include newest research findings to improve its accuracy and to respond to the evolving needs of a very diverse user community. In 2014, the G4-Med benchmarking system was born from the effort of the Geant4 Medical Simulation Benchmarking Group, to benchmark and monitor the evolution of Geant4 for medical physics applications. The G4-Med system was first described in our Medical Physics Special Report published in 2021. Results of the tests were reported for Geant4 10.5. Purpose In this work, we describe the evolution of the G4-Med benchmarking system. Methods The G4-Med benchmarking suite currently includes 23 tests, which benchmark Geant4 from the calculation of basic physical quantities to the simulation of more clinically relevant set-ups. New tests concern the benchmarking of Geant4-DNA physics and chemistry components for regression testing purposes, dosimetry for brachytherapy with a 125 I source, dosimetry for external x-ray and electron FLASH radiotherapy, experimental microdosimetry for proton therapy, and in vivo PET for carbon and oxygen beams. Regression testing has been performed between Geant4 10.5 and 11.1. Finally, a simple Geant4 simulation has been developed and used to compare Geant4 EM physics constructors and physics lists in terms of execution times. Results In summary, our EM tests show that the parameters of the multiple scattering in the Geant4 EM constructor G4EmStandardPhysics_option3 in Geant4 11.1, while improving the modeling of the electron backscattering in high atomic number targets, are not adequate for dosimetry for clinical x-ray and electron beams. Therefore, these parameters have been reverted back to those of Geant4 10.5 in Geant4 11.2.1. The x-ray radiotherapy test shows significant differences in the modeling of the bremsstrahlung process, especially between G4EmPenelopePhysics and the other constructors under study (G4EmLivermorePhysics, G4EmStandardPhysics_option3, and G4EmStandardPhysics_option4). These differences will be studied in an in-depth investigation within our Group. Improvement in Geant4 11.1 has been observed for the modeling of the proton and carbon ion Bragg peak with energies of clinical interest, thanks to the adoption of ICRU90 to calculate the low energy proton stopping powers in water and of the Linhard–Sorensen ion model, available in Geant4 since version 11.0. Nuclear fragmentation tests of interest for carbon ion therapy show differences between Geant4 10.5 and 11.1 in terms of fragment yields. In particular, a higher production of boron fragments is observed with Geant4 11.1, leading to a better agreement with reference data for this fragment. Conclusions Based on the overall results of our tests, we recommend to use G4EmStandardPhysics_option4 as EM constructor and QGSP_BIC_HP with G4EmStandardPhysics_option4, for hadrontherapy applications. The Geant4-DNA physics lists report differences in modeling electron interactions in water, however, the tests have a pure regression testing purpose so no recommendation can be formulated.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Report on G4-Med, a Geant4 benchmarking system for medical physics applications developed by the Geant4 Medical Simulation Benchmarking Group

Geant4 is a Monte Carlo code extensively used in medical physics for a wide range of applications, such as dosimetry, micro- and nanodosimetry, imaging, radiation protection, and nuclear medicine. Geant4 is continuously evolving, so it is crucial to have a system that benchmarks this Monte Carlo code for medical physics against reference data and to perform regression testing. In this work, to respond to these needs, we developed G4-Med, a benchmarking and regression testing system of Geant4 for medical physics. G4-Med currently includes 18 tests. They range from the benchmarking of fundamental physics quantities to the testing of Monte Carlo simulation setups typical of medical physics applications. Both electromagnetic and hadronic physics processes and models within the prebuilt Geant4 physics lists are tested. The tests included in G4-Med are executed on the CERN computing infrastructure via the use of the geant-val web application, developed at CERN for Geant4 testing. The physical observables can be compared to reference data for benchmarking and to results of previous Geant4 versions for regression testing purposes. This paper describes the tests included in G4-Med and shows the results derived from the benchmarking of Geant4 10.5 against reference data.

60 APPLIED LIFE SCIENCES↗

Code Coverage Status of the ARC Code RCT

The Argonne Reactor Code (ARC) software system supports users in their fast reactor design goals by providing neutronic, thermal-hydraulic, and structural analysis capabilities. REBUS plays a pivotal role in the ARC system as the primary fuel cycle analysis capability for fast reactor problems. Over its 60 year history, ARC software usage with REBUS has been applied to numerous fast and thermal spectrum reactor analysis projects with good to excellent comparison against experiments. The RCT code is a later addition and uses the REBUS restart files to define its input. The RCT code was built to provide pin depletion details on EBR-II models and thus many features of RCT were specifically tailored to the needs of EBR-II models. Additional approximations were invoked which are likely only valid for the EBR-II reactor and the particular fuel management that was done for it. The purpose of the present work is to identify a set of test problems for RCT and assess the code coverage for those test problems. The goal is to document what parts of the existing RCT code are touched by the set of test problems and which are not. Because no detailed verification work has been done on RCT, the existing regression testing suite was chosen for the code coverage assessment. The code coverage analysis of RCT was performed with the Code Coverage Tool of the Intel Fortran compiler which requires modifications to the compilation of RCT. The detailed coverage tables are given for each part of RCT. As will be discussed and shown, some parts of the RCT capability that are known to be used by the EBR-II analysis work are not tested by the regression testing suite. These aspects should be resolved before major source code changes are taken for the RCT software. Because REBUS and DIF3D are not subroutines of RCT, the coverage changes in both of those codes is not altered by RCT. The same is true for all of the modules of DIF3D that are used by RCT such as SYSLIB and SEGLIB.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

OLCF Test Harness

Acceptance and regression testing of a High Performance Computing (HPC) system requires an automated and reproducible framework and tool for running and logging results. Manually running tests across a system is labor intensive and prone to reproducibility errors. The OLCF Test Harness (OTH) provides a framework in which to document required tests for a HPC system. The OTH then provides tools to execute and log results of these tests in an automated fashion.

Dietz, Dan [Oak Ridge National Laboratory (ORNL), ↗

Current Status of SAMMY [Slides]

SAMMY is used to analyze experimental data mainly in the resolved range to generate evaluated data files for use in transport codes. The code base in SAMMY is still mostly F77, i.e., fixed line length and almost no use of modules. SAMMY is used in production environments, so during the modernization its functionality cannot be lost. That means we cannot simply rewrite it from scratch. Modernization is needed to allow us to add new features (for example new and improved algorithms in the resolved range). SAMMY has a large suite of regression tests that can be run and whose outputs can be checked automatically. During the modernization process, we will not change the results of these regression tests, except if errors in the original coding are found and corrected. Future plans for SAMMY include converting all container array access to Fortran allocate (or use C++ storage) and finishing the conversion to use C++ storage for all resonance parameters. We have an initial C++ implementation of the cross section reconstruction from R-Matrix parameters, used in AMPX. We will update this code, add reconstruction of angular data and derivatives and will use it in AMPX and SAMMY. Once this is done, we can start to add new features to the R-Matrix algorithm, like complex channel radii or relativistic effects. We also intend to modularize SAMMY to separate cross section reconstruction, experimental effect etc. from the Bayes fitting routines.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Fiscal Year 2024 Software Quality Assurance Activities for the ARC Software

The continued goal of the ARC SQA project in the Advanced Reactor Technologies program of DOE is to resolve the QA gaps for the ARC software that limit, or prevent, commercialization of the software for industry users. This project started in earnest in fiscal year 2023 which saw the entire code system moved from a SVN repository to a GitLab repository and an associated software quality assurance plan (SQAP) developed and ratified. Most of the QA gaps in the ARC software were identified in collaboration with industry partners and work begin in fiscal year 2023 and continued in 2024. The primary documentation that is missing includes user manuals, user guides, software verification reports, and code coverage assessments. The SUMMAR manual was completed this fiscal year and work was started on creating manuals for SE2ANL, SE2RCT, and DASSH. Software verification work was carried out for DIF3D and REBUS in a previous program and the current fiscal year saw the completion of software verification reports for GAMSOR, GAMSRC, VARPOW, EvaluateFlux, and SUMMAR. The goal for the next fiscal year is to complete the PERSENT software verification work and begin planning the software verification work for DASSH, SE2ANL, and SE2RCT. The code coverage reports for DIF3D and MC2-3 were completed in the previous fiscal year and the goal is to generate code coverage reports for REBUS, GAMSOR, PERSENT, and DASSH in the coming fiscal year. A considerable amount of effort was spent in the current fiscal year working on the continuous integration capability for automated regression testing in GitLab. The first version of the testing was created in the previous fiscal year and applied to DIF3D and its utility programs. That testing was extended this year to cover GAMSOR, REBUS, and PERSENT. To accomplish this, the first version of the new testing methodology had to be updated to make a single output checking methodology viable for all of the ARC software. This will result in a single document to detail the automated regression testing methodology and minor documents to detail the tolerance settings that have been applied to the output for each ARC code. The previous methodology put into place with SVN would have required a separate document for each ARC code to detail the output checking methodology and the tolerance settings for the output from each code. Because some of our industry partners are providing funds to add new capabilities to the ARC software to meet their needs, all of which must be reviewed and approved by the SQA program funded by this project, a summary of that development work is detailed in this report. Overall progress on resolving the QA gaps has been good this year with the most impactful improvement for our industry partners in capability being the creation of a threaded version of DIF3D-VARIANT that allows the DIF3D, REBUS, and GAMSOR run times to be reduced by a factor of 4-6. The most impactful QA gap that was resolved was the software verification of GAMSRC and VARPOW.

97 MATHEMATICS AND COMPUTING↗