Search NASA⌕ Search

SEARCH · Search NASA

Results for “verification & validation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
NASA NTRS2005

Modeling and Simulation Verification, Validation and Accreditation (VV&A): A New Undertaking for the Exploration Systems Mission Directorate

and Accreditation (VV&A) session audience, a snapshot review of the Exploration Space Mission Directorate s (ESMD) investigation into implementation of a modeling and simulation (M&S) VV&A program. The presentation provides some legacy ESMD reference material, including information on the then-current organizational structure, and M&S (Simulation Based Acquisition (SBA)) focus contained therein, to provide a context for the proposed M&S VV&A approach. This reference material briefly highlights the SBA goals and objectives, and outlines FY05 M&S development and implementation consistent with the Subjective Assessment, Constructive Assessment, Operator-in-the-Loop Assessment, Hardware-in-the-Loop Assessment, and In Service Operations Assessment M&S construct, the NASA Exploration Information Ontology Model (NExIOM) data model, and integration with the Windchill-based Integrated Collaborative Environment (ICE). The presentation then addresses the ESMD team s initial conclusions regarding an M&S VV&A program, summarizes the general VV&A implementation approach anticipated, and outlines some of the recognized VV&A program challenges, all within a broader context of the overarching Integrated Modeling and Simulation (IM&S) environment at both the ESMD and Agency (NASA) levels. The presentation concludes with a status on the current M&S organization s progress to date relative to the recommended IM&S implementation activity. The overall presentation was focused to provide, for the Verification, Validation,

Prill, Mark E.↗
DOE OSTI2017

VERA-CS Verification & Validation Plan

This report summarizes the current status of VERA-CS Verification and Validation for PWR Core Follow operation and proposes a multi-phase plan for continuing VERA-CS V&V in FY17 and FY18. The proposed plan recognizes the hierarchical nature of a multi-physics code system such as VERA-CS and the importance of first achieving an acceptable level of V&V on each of the single physics codes before focusing on the V&V of the coupled physics solution. The report summarizes the V&V of each of the single physics codes systems currently used for core follow analysis (ie MPACT, CTF, Multigroup Cross Section Generation, and BISON / Fuel Temperature Tables) and proposes specific actions to achieve a uniformly acceptable level of V&V in FY17. The report also recognizes the ongoing development of other codes important for PWR Core Follow (e.g. TIAMAT, MAMBA3D) and proposes Phase II (FY18) VERA-CS V&V activities in which those codes will also reach an acceptable level of V&V. The report then summarizes the current status of VERA-CS multi-physics V&V for PWR Core Follow and the ongoing PWR Core Follow V&V activities for FY17. An automated procedure and output data format is proposed for standardizing the output for core follow calculations and automatically generating tables and figures for the VERA-CS Latex file. A set of acceptance metrics is also proposed for the evaluation and assessment of core follow results that would be used within the script to automatically flag any results which require further analysis or more detailed explanation prior to being added to the VERA-CS validation base. After the Automation Scripts have been completed and tested using BEAVRS, the VERA-CS plan proposes the Watts Bar cycle depletion cases should be performed with the new cross section library and be included in the first draft of the new VERA-CS manual for release at the end of PoR15. Also, within the constraints imposed by the proprietary nature of plant data, as many as possible of the FY17 AMA Plant Core Follow cases should also be included in the VERA-CS manual at the end of PoR15. After completion of the ongoing development of TIAMAT for fully coupled, full core calculations with VERA-CS / BISON 1.5D, and after the completion of the refactoring of MAMBA3D for CIPS analysis in FY17, selected cases from the VERA-CS validation based should be performed, beginning with the legacy cases of Watts Bar and BEAVRS in PoR16. Finally, as potential Phase III future work some additional considerations are identified for extending the VERA-CS V&V to other reactor types such as the BWR.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
NASA NTRS2014

Integrated Medical Model Verification, Validation, and Credibility

The Integrated Medical Model (IMM) was designed to forecast relative changes for a specified set of crew health and mission success risk metrics by using a probabilistic (stochastic process) model based on historical data, cohort data, and subject matter expert opinion. A probabilistic approach is taken since exact (deterministic) results would not appropriately reflect the uncertainty in the IMM inputs. Once the IMM was conceptualized, a plan was needed to rigorously assess input information, framework and code, and output results of the IMM, and ensure that end user requests and requirements were considered during all stages of model development and implementation. METHODS: In 2008, the IMM team developed a comprehensive verification and validation (VV) plan, which specified internal and external review criteria encompassing 1) verification of data and IMM structure to ensure proper implementation of the IMM, 2) several validation techniques to confirm that the simulation capability of the IMM appropriately represents occurrences and consequences of medical conditions during space missions, and 3) credibility processes to develop user confidence in the information derived from the IMM. When the NASA-STD-7009 (7009) was published, the IMM team updated their verification, validation, and credibility (VVC) project plan to meet 7009 requirements and include 7009 tools in reporting VVC status of the IMM. RESULTS: IMM VVC updates are compiled recurrently and include 7009 Compliance and Credibility matrices, IMM VV Plan status, and a synopsis of any changes or updates to the IMM during the reporting period. Reporting tools have evolved over the lifetime of the IMM project to better communicate VVC status. This has included refining original 7009 methodology with augmentation from the NASA-STD-7009 Guidance Document. End user requests and requirements are being satisfied as evidenced by ISS Program acceptance of IMM risk forecasts, transition to an operational model and simulation tool, and completion of service requests from a broad end user consortium including Operations, Science and Technology Planning, and Exploration Planning. CONCLUSIONS: The VVC approach established by the IMM project of combining the IMM VV Plan with 7009 requirements is comprehensive and includes the involvement of end users at every stage in IMM evolution. Methods and techniques used to quantify the VVC status of the IMM have not only received approval from the local NASA community but have also garnered recognition by other federal agencies seeking to develop similar guidelines in the medical modeling community.

Validation↗
NASA NTRS2015

Integrated Medical Model (IMM) Project Verification, Validation, and Credibility (VVandC)

The Integrated Medical Model (IMM) Project supports end user requests by employing the Integrated Medical Evidence Database (iMED) and IMM tools as well as subject matter expertise within the Project. The iMED houses data used by the IMM. The IMM is designed to forecast relative changes for a specified set of crew health and mission success risk metrics by using a probabilistic model based on historical data, cohort data, and subject matter expert opinion. A stochastic approach is taken because deterministic results would not appropriately reflect the uncertainty in the IMM inputs. Once the IMM was conceptualized, a plan was needed to rigorously assess input information, framework and code, and output results of the IMM, and ensure that end user requests and requirements were considered during all stages of model development and implementation, as well as lay the foundation for external review and application. METHODS: In 2008, the Project team developed a comprehensive verification and validation (VV) plan, which specified internal and external review criteria encompassing 1) verification of data and IMM structure to ensure proper implementation of the IMM, 2) several validation techniques to confirm that the simulation capability of the IMM appropriately represents occurrences and consequences of medical conditions during space missions, and 3) credibility processes to develop user confidence in the information derived from the IMM. When the NASA-STD-7009 (7009) [1] was published, the Project team updated their verification, validation, and credibility (VVC) project plan to meet 7009 requirements and include 7009 tools in reporting VVC status of the IMM. Construction of these tools included meeting documentation and evidence requirements sufficient to meet external review success criteria. RESULTS: IMM Project VVC updates are compiled recurrently and include updates to the 7009 Compliance and Credibility matrices. Reporting tools have evolved over the lifetime of the IMM Project to better communicate VVC status. This has included refining original 7009 methodology with augmentation from the HRP NASA-STD-7009 Guidance Document working group and the NASA-HDBK-7009 [2]. End user requests and requirements are being satisfied as evidenced by ISS Program acceptance of IMM risk forecasts, transition to an operational model and simulation tool, and completion of service requests from a broad end user consortium including operations, science and technology planning, and exploration planning. IMM v4.0 is slated for operational release in the FY015 and current VVC assessments illustrate the expected VVC status prior to the completion of customer lead external review efforts. CONCLUSIONS: The VVC approach established by the IMM Project of incorporating Project-specific recommended practices and guidelines for implementing the 7009 requirements is comprehensive and includes the involvement of end users at every stage in IMM evolution. Methods and techniques used to quantify the VVC status of the IMM Project represented a critical communication tool in providing clear and concise suitability assessments to IMM customers. These processes have not only received approval from the local NASA community but have also garnered recognition by other federal agencies seeking to develop similar guidelines in the medical modeling community.

Walton, M.↗
DOE OSTI2019

Progress Towards A New CTH Code Verification & Validation Test Suite

The CTH multiphysics hydrocode is used in a wide variety of important calculations. An essential part of ensuring hydrocode accuracy and credibility is thorough code verification and validation (V&V). In the past, CTH V&V work (particularly verification) has not been consistently well documented. In FY19, we have made substantial progress towards addressing this need. In this report, we present a new CTH V&V test suite composed of traditional hydrocode verification problems used by similar ASC codes as well as validation problems for some of the most frequently used materials models and capabilities in CTH. For the verification problems, we present not only results and computed errors, but also convergence rates. Validation problems include mesh refinement studies, providing evidence that results are converging.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2013

Verification & Validation of High-Order Short-Characteristics-Based Deterministic Transport Methodology on Unstructured Grids

The research team has developed a practical, high-order, discrete-ordinates, short characteristics neutron transport code for three-dimensional configurations represented on unstructured tetrahedral grids that can be used for realistic reactor physics applications at both the assembly and core levels. This project will perform a comprehensive verification and validation of this new computational tool against both a continuous-energy Monte Carlo simulation (e.g. MCNP) and experimentally measured data, an essential prerequisite for its deployment in reactor core modeling. Verification is divided into three phases. The team will first conduct spatial mesh and expansion order refinement studies to monitor convergence of the numerical solution to reference solutions. This is quantified by convergence rates that are based on integral error norms computed from the cell-by-cell difference between the code’s numerical solution and its reference counterpart. The latter is either analytic or very fine- mesh numerical solutions from independent computational tools. For the second phase, the team will create a suite of code-independent benchmark configurations to enable testing the theoretical order of accuracy of any particular discretization of the discrete ordinates approximation of the transport equation. For each tested case (i.e. mesh and spatial approximation order), researchers will execute the code and compare the resulting numerical solution to the exact solution on a per cell basis to determine the distribution of the numerical error. The final activity comprises a comparison to continuous-energy Monte Carlo solutions for zero-power critical configuration measurements at Idaho National Laboratory’s Advanced Test Reactor (ATR). Results of this comparison will allow the investigators to distinguish between modeling errors and the above-listed discretization errors introduced by the deterministic method, and to separate the sources of uncertainty.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
NASA NTRS2024

Validation & Verification of CFD Models for Cryogenic Fluid Management of Propellant Tanks in Space

This article describes the need and the strategy for CFD model development, validation, and verification for Cryogenic Fluid Management (CFM) of Propellant Tanks in Space. It describes the type of CFD models that must be developed to address the future needs of Space CFM. It also discusses the two classes of experiments currently used to validate the fidelity of the CFD models. These two experiment classes are: (a) the small-scale simulant fluid science experiments that are equipped with scientific diagnostics to elucidate the underlying two-phase fluid physics of the CFM processes; and (b) the large-scale cryogenic experiments that assess the engineering performance of the propellant tank for storage and transfer. The current status of the CFD model development and validation is briefly assessed by presenting examples of segregated two-phase flow problems that have been successfully modeled. The future model development directions for CFM situations involving more complex interpenetrating phases are also defined.

Cryogenic Fluid Management↗
DOE OSTI2015

SU-E-T-524: In-Vivo Diode Dosimetry Proton Therapy Range Verification Validation Study for Pediatric CSI

Purpose: With the aim of reducing acute esophageal radiation toxicity in pediatric patients receiving craniospinal irradiation (CSI), we investigated the implementation of an in-vivo, adaptive proton therapy range verification methodology. Simulation experiments and in-phantom measurements were conducted to validate the range verification technique for this clinical application. Methods: A silicon diode array system has been developed and experimentally tested in phantom for passively scattered proton beam range verification for a prostate treatment case by correlating properties of the detector signal to the water equivalent path length (WEPL). We propose to extend the methodology to verify range distal to the vertebral body for pediatric CSI cases by placing this small volume dosimeter in the esophagus of the anesthetized patient immediately prior to treatment. A set of calibration measurements was performed to establish a time signal to WEPL fit for a “scout” beam in a solid water phantom. Measurements are compared against Monte Carlo simulation in GEANT4 using the Tool for Particle Simulation (TOPAS). Results: Measurements with the diode array in a spread out Bragg peak of 14 cm modulation width and 15 cm range (177 MeV passively scattered beam) in solid water were successfully validated against proton fluence rate simulations in TOPAS. The resulting calibration curve allows for a sensitivity analysis of detector system response with dose rate in simulation and with individual diode position through simulation on patient CT data. Conclusion: Feasibility has been shown for the application of this range verification methodology to pediatric CSI. An in-vivo measurement to determine the WEPL to the inner surface of the esophagus will allow for personalized adjustment of the treatment plan to ensure sparing of the esophagus while confirming target coverage. A Toltz acknowledges partial support by the CREATE Medical Physics Research Training Network grant of the Natural Sciences and Engineering Research Council (Grant number: 432290)

60 APPLIED LIFE SCIENCES↗
NASA NTRS2019

Psyche Early Project Verification & Validation Planning Development

The Psyche mission to the asteroid (16) Psyche was selected as the fourteenth mission in the Discovery program in January 2017. The Psyche mission will determine if (16) Psyche is the core of a larger differentiated body. As part of the development of this mission a Verification and Validation (V&V) engineer was assigned early in the project’s design Phase B. This paper will discuss some of the strategies that the Psyche team is using to take full advantage of the early planning for V&V on flight projects and specifically how the Psyche mission is approaching these tasks. This paper will discuss the effects of having a V&V mindset on: 1) The Psyche requirements development process, and how focusing not only on how the team will verify these requirements but also on developing the tools necessary to track and monitor that verification feeds back into the requirement development process. 2) The Psyche testbed development, and how using a verification mindset is useful for identifying holes in the testbed development process, including the required testbed speed and how to think about testbed certification early in the process while encouraging trades and developing relationships with the testbed team. 3) Developing new V&V Tools for Psyche, and how developing tools early in the project development process means that they can influence the development of other requirement and scheduling tools. 4) Verification Activity Planning, which is typically done to a preliminary level during Phase B of the project, and is important for understanding the major testing that is needed to ensure that the system as built represents the design. On Psyche we are not only focused on bringing this planning to a preliminary level, we are also developing a V&V focused schedule to help us de-conflict V&V activities that may require similar resources early on in the program’s development. 5) Validation Planning, for which we are working with the testbed and model development teams to ensure that their models can be effectively validated and that the plans to do so are in place. In this paper we will describe how the Psyche mission is approaching each of these V&V areas and identify lessons that can be taken by other space missions trying to decide how much effort should be put into V&V early in the project lifecycle.

Solish, Benjamin↗
NASA NTRS2023

Autonomy Verification & Validation Roadmap and Vision 2045

Advanced capabilities planned for the next generation of autonomous and increasingly autonomous air vehicles will include non-traditional components based on artificial intelligence, machine learning, and complex optimization and planning algorithms. These complex components will be used to provide enhanced safety and high-level decision-making functions. However, there are serious barriers to the deployment of autonomous aircraft in the National Airspace System (NAS). Current civil aviation certification processes are based on the concept that the correct behavior of a system or a component must be completely specified and verified prior to operation. This report from the Autonomy Verification and Validation (V&V) Roadmap and Vision 2045 project presents the most recent effort to build a comprehensive list of verification challenges and needs for autonomous aircraft, a roadmap to meet those autonomy V&V needs, the services they can enable, and point to the certification gaps they fill. To accomplish these goals, we assembled a team of world-class researchers from the aerospace industry (Boeing, Collins Aerospace, and GeneralElectric) and academia (University of Michigan, University of Texas, and Massachusetts Institute of Technology) with deep expertise in autonomy, aerospace systems, and assurance of Artificial Intelligence/machine learning systems.

Software Assurance↗
DOE OSTI2022

Verification, Validation, and Uncertainty Quantification in Thermal Hydraulics, Freeman Scholar Lecture (2019)

Engineering problems are generally solved by analytical models or computer codes. These models, in addition to conservation equations, also include many empirical relationships and approximate numerical methods. Each of these components contributes to the uncertainty in the prediction. A systematic approach to judge the applicability of the code to the intended application is needed. It starts from verification of implementation of formulation in the code, identification of important phenomena, finding relevant tests with quantified uncertainty for these phenomena, and validation of the code by comparing predictions with the relevant test data. The relevant tests must address phenomena as expected in the intended application. In case of small size or limited condition tests, the scaling analyses are needed to assess the relevancy of the tests. Finally, a statement of uncertainty in the prediction is needed. Systematic approaches are described to aggregate uncertainties from different components of the code for intended application. Here, verification, validation, and uncertainty quantifications (VVUQs) are briefly described.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2022

FY2021 Improvements to the New CTH Code Verification & Validation Test Suite

Over the past few years, the CTH multiphysics hydrocode has overhauled its software quality and testing processes, implementing current best practices in software quality and building a robust V&V test suite comprised of traditional hydrocode verification problems, including ASC Tri-Lab Test Suite and Enhanced Tri-Lab Test Suite problems, as well as validation problems for some of CTH’s most frequently used equations of state, materials models, and other key capabilities. Substantial progress towards building this new test suite was made in FY19 and FY20. In FY21, the test suite has been expanded to include verification and validation tests of the Steinberg-Guinan-Lund (ST) viscoplastic model and the Johnson Cook (JFRAC) fracture model. Additionally, two new verification tests were added, covering hydrodynamics and high explosive (HE) modeling capabilities: the Kidder Gaussian density problem and the Escape of HE Products (EHEP) problem from the Tri-Lab Test Suite. This report discusses each of these test problems in detail. Verification test results are compared to analytic solutions. Validation test results are compared to experimental data. Wherever possible, convergence or mesh refinement studies are included. Additionally, while implementing the Kidder verification problem, a bug was identified that affects the use of tables to initialize pressure or density in 1D or 2D calculations. A brief discussion of the bug and its fix is included. CTH demonstrates good performance overall on the new test suite problems. Simulation results showed good agreement with analytic solutions for the Kidder problem, with convergence rates ranging between 1.8 and sub-linear, and relatively good agreement for the EHEP problem, though convergence rates for pressure and density were nearly 0. The ST and JFRAC strain rate loading verification tests show good agreement with analytic solutions. Likewise, CTH simulation results show good agreement with experimental validation data, including Taylor rod impact testing, for the materials tested. Future V&V work will focus on adding 2D and 3D versions of existing verification tests as well as adding validation tests of other frequently used capabilities such as other fracture models.

42 ENGINEERING↗
DOE OSTI2020

Improvements to the New CTH Code Verification & Validation Test Suite (FY2020)

The CTH multiphysics hydrocode, which is used for a wide range of important calculations, has undertaken in recent years to overhaul its software quality and testing processes. A key part of this effort entailed building a new, robust V&V test suite made up of traditional hydrocode verification problems, such as those listed in the ASC Tri-Lab Test Suite and the Enhanced Tri-Lab Test Suite, as well as validation problems for some of CTHs most frequently used equations of state, materials models, and other key capabilities. Substantial progress towards this goal was made in FY19. In FY20, this test suite has been expanded to include verification and validation tests of the Sesame and JWL equation of state models as well as the Mader verification problem from the Tri-Lab Test Suite and the Blake verification problem - a linear elastic analog to the Hunter problem from the Enhanced Tri-Lab Test Suite. This report documents CTH performance on the new test suite problems. Verification test results are compared to analytic solutions and, for most tests, convergence results are presented. Validation test results are compared to experimental data and mesh refinement studies are included. CTH performs well overall on the new test problems. Convergence rates for the Blake and Mader problems are comparable to those for similar ASC codes. The JWL and Sesame verification tests show good agreement with analytic solutions. Likewise, CTH simulation results show good agreement with experimental validation data for the Sesame and JWL equations of state for the materials tested. Future V&V work will focus on adding tests for other key capabilities like fracture and high explosive models.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
NASA NTRS2006

Sensor to User - NASA/EOS Data for Coastal Zone Management Applications Developed from Integrated Analyses: Verification, Validation and Benchmark Report

The NASA Applied Sciences Program seeks to transfer NASA data, models, and knowledge into the hands of end-users by forming links with partner agencies and associated decision support tools (DSTs). Through the NASA REASoN (Research, Education and Applications Solutions Network) Cooperative Agreement, the Oceanography Division of the Naval Research Laboratory (NRLSSC) is developing new products through the integration of data from NASA Earth-Sun System assets with coastal ocean forecast models and other available data to enhance coastal management in the Gulf of Mexico. The recipient federal agency for this research effort is the National Oceanic and Atmospheric Administration (NOAA). The contents of this report detail the effort to further the goals of the NASA Applied Sciences Program by demonstrating the use of NASA satellite products combined with data-assimilating ocean models to provide near real-time information to maritime users and coastal managers of the Gulf of Mexico. This effort provides new and improved capabilities for monitoring, assessing, and predicting the coastal environment. Coastal managers can exploit these capabilities through enhanced DSTs at federal, state and local agencies. The project addresses three major issues facing coastal managers: 1) Harmful Algal Blooms (HABs); 2) hypoxia; and 3) freshwater fluxes to the coastal ocean. A suite of ocean products capable of describing Ocean Weather is assembled on a daily basis as the foundation for this semi-operational multiyear effort. This continuous realtime capability brings decision makers a new ability to monitor both normal and anomalous coastal ocean conditions with a steady flow of satellite and ocean model conditions. Furthermore, as the baseline data sets are used more extensively and the customer list increased, customer feedback is obtained and additional customized products are developed and provided to decision makers. Continual customer feedback and response with new improved products are required between the researcher and customer. This document details the methods by which these coastal ocean products are produced including the data flow, distribution, and verification. Product applications and the degree to which these products are used successfully within NOAA and coordinated with the Mississippi Department of Marine Resources (MDMR) is benchmarked.

Hall, Callie↗
DOE OSTI2024

CI/CD Efforts for Validation, Verification and Benchmarking OpenMP Implementations

Software developers must adapt to keep up with the changing capabilities of platforms so that they can utilize the power of High-Performance Computers (HPC), including exascale systems. OpenMP, a directive-based parallel programming model, allows developers to include directives to existing C, C++, or Fortran code to allow node level parallelism without compromising performance. This paper describes our CI/CD efforts to provide easy evaluation of the support of OpenMP across different compilers using existing testsuites and benchmark suites on HPC platforms. Our main contributions include (1) the set of a Continuous Integration (CI) and Continuous Development (CD) workflow that captures bugs and provides faster feedback to compiler developers, (2) an evaluation of OpenMP (offloading) implementations supported by AMD, HPE, GNU, LLVM, and Intel, and (3) evaluation of the quality of compilers across different heterogeneous HPC platforms. With the comprehensive testing through the CI/CD workflow, we aim to provide a comprehensive understanding of the current state of OpenMP (offloading) support in different compilers and heterogeneous platforms consisting of CPUs and GPUs from NVIDIA, AMD, and Intel.

Jarmusch, Aaron↗