Exploitation of a Validation Hierarchy for Modeling and Simulation

Scott Shaw; James M Luckring; William L Oberkampf; Rick E Graves

NASA NTRS · 20220014628

Exploitation of a Validation Hierarchy for Modeling and Simulation

Scott Shaw·James M Luckring·William L Oberkampf·Rick E Graves

Abstract

Across engineering there is an evolving need to increase reliance on physics-based simulation to develop, design and optimize engineering systems. This increased reliance on modeling and simulation has highlighted a growing need to transform the confidence that modeling and simulation analysts have in their results into credibility for systems engineers to design and field systems more quickly and with less physical testing. For isolated components of a complex system, where a single discipline may drive product design, this is less of a concern as the relationship is often straightforward and easy to explain. However, when these isolated components are integrated, and are expected to operate in a multi-disciplinary context in which safety critical systems are involved, new concepts and model assurance standards are required. In this paper we address this challenge by showing how a model validation hierarchy can be exploited to identify those model validation experiments that will contribute most to increasing confidence and credibility of modeling and simulation predictions. The approach that is adopted contains four main steps. The first step is the construction of a model validation hierarchy that links subsystems, assemblies, and components to a hierarchy of physical experiments that can be used support model validation. This hierarchy connects the concerns of systems engineers to those of the modeling and simulation analyst in a clear and logical way. The structure and content of this hierarchy is then used in a second step to establish which physical phenomena have the greatest impact on overall system performance metrics. A gap analysis technique, based upon modeling and simulation concerns, is then used to prioritize the important physical phenomenon. Unfortunately, a common outcome of such gap analyses is the identification of many important gaps and so, in the final step of our process, we advocate the use of a global sensitivity analysis as a means to complete the prioritization.

Explore related subjects

Verification and Validation Modeling and Simulation Validation Hierarchy Missiles

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Scott Shaw, James M Luckring, William L Oberkampf, Rick E Graves. Exploitation of a Validation Hierarchy for Modeling and Simulation. https://ntrs.nasa.gov/citations/20220014628

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related reports

NASA NTRS2020

Adaptive Stress Testing: Finding Likely Failure Events with Reinforcement Learning

Finding the most likely path to a set of failure states is important to the analysis of safety-critical systems that operate over a sequence of time steps, such as aircraft collision avoidance systems and autonomous cars. In many applications such as autonomous driving, failures cannot be completely eliminated due to the complex stochastic environment in which the system operates.As a result, safety validation is not only concerned about whether a failure can occur, but also discovering which failures are most likely to occur. This article presents adaptive stress testing (AST), a framework for finding the most likely path to a failure event in simulation. We consider a general black box setting for partially observable and continuous-valued systems operating in an environment with stochastic disturbances. We formulate the problem as a Markov decision process and use reinforcement learning to optimize it. The approach is simulation-based and does not require internal knowledge of the system, making it suitable for black-box testing of large systems. We present different formulations depending on whether the state is fully observable or partially observable. In the latter case, we present a modified Monte Carlo tree search algorithm that only requires access to the pseudorandom number generator of the simulator to overcome partial observability. We also present an extension of the framework, called differential adaptive stress testing (DAST), that can find failures that occur in one system but not in another. This type of differential analysis is useful in applications such as regression testing, where we are concerned with finding areas of relative weakness compared to a baseline. We demonstrate the effectiveness of the approach on an aircraft collision avoidance application, where a prototype aircraft collision avoidance system is stress tested to find the most likely scenarios of near mid-air collision.

Verification and Validation↗

NASA NTRS2015

Adaptive Stress Testing of Airborne Collision Avoidance Systems

This paper presents a scalable method to efficiently search for the most likely state trajectory leading to an event given only a simulator of a system. Our approach uses a reinforcement learning formulation and solves it using Monte Carlo Tree Search (MCTS). The approach places very few requirements on the underlying system, requiring only that the simulator provide some basic controls, the ability to evaluate certain conditions, and a mechanism to control the stochasticity in the system. Access to the system state is not required, allowing the method to support systems with hidden state. The method is applied to stress test a prototype aircraft collision avoidance system to identify trajectories that are likely to lead to near mid-air collisions. We present results for both single and multi-threat encounters and discuss their relevance. Compared with direct Monte Carlo search, this MCTS method performs significantly better both in finding events and in maximizing their likelihood.

Verification and Validation↗

NASA NTRS2015

Rapid State Space Modeling Tool for Rectangular Wing Aeroservoelastic Studies

This report introduces a modeling and simulation tool for aeroservoelastic analysis of rectangular wings with trailing-edge control surfaces. The inputs to the code are planform design parameters such as wing span, aspect ratio, and number of control surfaces. Using this information, the generalized forces are computed using the doublet-lattice method. Using Roger's approximation, a rational function approximation is computed. The output, computed in a few seconds, is a state space aeroservoelastic model which can be used for analysis and control design. The tool is fully parameterized with default information so there is little required interaction with the model developer. All parameters can be easily modified if desired. The focus of this report is on tool presentation, verification, and validation. These processes are carried out in stages throughout the report. The rational function approximation is verified against computed generalized forces for a plate model. A model composed of finite element plates is compared to a modal analysis from commercial software and an independently conducted experimental ground vibration test analysis. Aeroservoelastic analysis is the ultimate goal of this tool, therefore, the flutter speed and frequency for a clamped plate are computed using damping-versus-velocity and frequency-versus-velocity analysis. The computational results are compared to a previously published computational analysis and wind-tunnel results for the same structure. A case study of a generic wing model with a single control surface is presented. Verification of the state space model is presented in comparison to damping-versus-velocity and frequency-versus-velocity analysis, including the analysis of the model in response to a 1-cos gust.

Verification and Validation↗