Search NASA⌕ Search

SEARCH · Search NASA

Results for “Verification & validation (V&V)”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
NASA NTRS2022

Actuator and Motor Control End-to-End V&V on the Mars 2020 Rover

The Mars 2020 Perseverance rover is the most advanced robotic exploration system ever sent to another planet. To support the complex scientific and mobility needs of the mission, the rover utilizes 33 actuators, three multi-degree-of-freedom force-torque sensors, fifteen single or dual-speed resolvers, two solenoid valves, and twelve contact switches. The control for these actuators and sensors is achieved by several levels of flight software, coordinated between two computers with varying bandwidth control loops. Furthermore, the actuators and sensors were integrated into multiple larger robotic mechanisms that were delivered by different organizations at various points in the Integration and Test (I&T) timeline. All of this created a very complex Verification and Validation (V&V) scenario involving multiple subsystems and teams, several hardware and software testbeds with varying levels of fidelity, and significant systems engineering to ensure the overall I&T schedule could be maintained while ensuring system hardware safety.This paper details the integrated V&V effort across multiple teams and venues to provide full coverage of all necessary functionality, performance, and fault protection. First, it provides an overview of how the V&V campaign was subdivided among teams and venues and provides descriptions of the various hardware configurations used to support the testing. The Mars 2020 implementation of the plan incorporates many of the lessons learned from Mars Science Laboratory’s test campaign, and these value-added modifications are discussed here. Also included in this section is the system-level environmental testing approach used for mechanisms. Second, the paper describes the phased approach used by the teams to support new hardware and software deliveries to testbed and Systems I&T. In this approach the test campaign was built upon higher-level mechanism needs for performance, functionality, and safety at specific times in the campaign. Finally, the paper discusses lessons learned from the V&V campaign that should be applied to future large-scale motion control testing efforts.

Borne, Davis↗
NASA NTRS2008

Assessing Requirements Quality through Requirements Coverage

In model-based development, the development effort is centered around a formal description of the proposed software system the model. This model is derived from some high-level requirements describing the expected behavior of the software. For validation and verification purposes, this model can then be subjected to various types of analysis, for example, completeness and consistency analysis [6], model checking [3], theorem proving [1], and test-case generation [4, 7]. This development paradigm is making rapid inroads in certain industries, e.g., automotive, avionics, space applications, and medical technology. This shift towards model-based development naturally leads to changes in the verification and validation (V&V) process. The model validation problem determining that the model accurately captures the customer's high-level requirements has received little attention and the sufficiency of the validation activities has been largely determined through ad-hoc methods. Since the model serves as the central artifact, its correctness with respect to the users needs is absolutely crucial. In our investigation, we attempt to answer the following two questions with respect to validation (1) Are the requirements sufficiently defined for the system? and (2) How well does the model implement the behaviors specified by the requirements? The second question can be addressed using formal verification. Nevertheless, the size and complexity of many industrial systems make formal verification infeasible even if we have a formal model and formalized requirements. Thus, presently, there is no objective way of answering these two questions. To this end, we propose an approach based on testing that, when given a set of formal requirements, explores the relationship between requirements-based structural test-adequacy coverage and model-based structural test-adequacy coverage. The proposed technique uses requirements coverage metrics defined in [9] on formal high-level software requirements and existing model coverage metrics such as the Modified Condition and Decision Coverage (MC/DC) used when testing highly critical software in the avionics industry [8]. Our work is related to Chockler et al. [2], but we base our work on traditional testing techniques as opposed to verification techniques.

Rajan, Ajitha↗
NASA NTRS2021

Europa Clipper Payload Verification and Validation: Early Architecture and Implementation

NASA’s Europa Clipper mission will investigate the Jovian icy moon Europa with a sophisticated payload consisting of a suite of nine instruments. Clipper’s observations will help provide answers to a broad range of scientific objectives related to Europa’s habitability. As the project proceeds through Phase C, we are building confidence in the system’s ability to achieve these science goals by implementing a rigorous payload verification and validation (V&V) program. This paper details the planned structure and implementation of the Europa Clipper payload V&V program as it exists leading up to the project critical design review (CDR), before the bulk of instrument- or payload- level V&V activities have begun. We first explain the context for payload V&V on Europa Clipper, including its interfaces with other parts of the project system. We then describe the taxonomy of V&V concepts and their interrelationships, followed by a discussion on how this framework allows us to codify interfaces, processes, and ideas that are key to a successful V&V program. We focus on key aspects of the planning phases of V&V, especially establishing a complete set of verification items and accurately mapping them into appropriate verification activities. The second half of the paper describes the logistics of implementing this framework onto our actual payload system, including the processes and tools we use to manage and communicate agreements, changes, and risks across the project. We conclude with a reflection on the unique drivers and challenges of the V&V program concept development so far and the work that is planned for phase D implementation.

Srivastava, Priyanka↗
NASA NTRS2018

Assurance of Model-Based Fault Diagnosis

Autonomy is an increasingly important technology for robotic scientific and commercial spacecraft. An important motivation for developing onboard autonomy is to enable quick response to dynamic environment and situations, including fault conditions that a spacecraft may encounter. The reliability of such autonomous capabilities depends on the quality of their knowledge of a spacecraft’s health state. Model-based approaches to fault management, i.e. model-based fault diagnosis (MBFD), is one approach to continuously verify correct behavior in addition to diagnosing symptoms to estimate the spacecraft’s health state. The proper functioning of MBFD is dependent on 1) the quality of the model that is analyzed and compared to the outputs of onboard sensors to estimate the system’s health state, and 2) the correct functioning of the diagnosis engine that interrogates the model and compares its analysis to observed system behavior. We are currently developing Verification and Validation (V&V) approaches to provide the necessary confidence that MBFD systems are correctly estimating the health of on-board spacecraft components and systems. Our work is intended to narrow the gap between the rapidly maturing field of ModelBased System Engineering (including MBFD) and the less-well understood area of identifying and applying appropriate V&V techniques to MBFD. Our effort is investigating three areas: 1) developing V&V techniques for the diagnostic model, 2) developing V&V techniques for the diagnostic engine in isolation, and 3) developing V&V techniques for the diagnostic engine and model in combination. This paper describes the work we have completed in the first area. We describe our approach to selecting a system to be represented by a model, the approach to modeling the system, the verification approach we developed for the model, and the results of the verification activity. We conclude with a description of the work remaining in the last two areas, which will be addressed over the next two years.

Chung, Seung↗
NASA NTRS2013

Model Verification and Validation Concepts for a Probabilistic Fracture Assessment Model to Predict Cracking of Knife Edge Seals in the Space Shuttle Main Engine High Pressure Oxidizer

Physics-based models are routinely used to predict the performance of engineered systems to make decisions such as when to retire system components, how to extend the life of an aging system, or if a new design will be safe or available. Model verification and validation (V&V) is a process to establish credibility in model predictions. Ideally, carefully controlled validation experiments will be designed and performed to validate models or submodels. In reality, time and cost constraints limit experiments and even model development. This paper describes elements of model V&V during the development and application of a probabilistic fracture assessment model to predict cracking in space shuttle main engine high-pressure oxidizer turbopump knife-edge seals. The objective of this effort was to assess the probability of initiating and growing a crack to a specified failure length in specific flight units for different usage and inspection scenarios. The probabilistic fracture assessment model developed in this investigation combined a series of submodels describing the usage, temperature history, flutter tendencies, tooth stresses and numbers of cycles, fatigue cracking, nondestructive inspection, and finally the probability of failure. The analysis accounted for unit-to-unit variations in temperature, flutter limit state, flutter stress magnitude, and fatigue life properties. The investigation focused on the calculation of relative risk rather than absolute risk between the usage scenarios. Verification predictions were first performed for three units with known usage and cracking histories to establish credibility in the model predictions. Then, numerous predictions were performed for an assortment of operating units that had flown recently or that were projected for future flights. Calculations were performed using two NASA-developed software tools: NESSUS(Registered Trademark) for the probabilistic analysis, and NASGRO(Registered Trademark) for the fracture mechanics analysis. The goal of these predictions was to provide additional information to guide decisions on the potential of reusing existing and installed units prior to the new design certification.

Pai, Shantaram S.↗
NASA NTRS1998

Towards a Theory for Integration of Mathematical Verification and Empirical Testing

From the viewpoint of a project manager responsible for the V&V (verification and validation) of a software system, mathematical verification techniques provide a possibly useful orthogonal dimension to otherwise standard empirical testing. However, the value they add to an empirical testing regime both in terms of coverage and in fault detection has been difficult to quantify. Furthermore, potential cost savings from replacing testing with mathematical verification techniques cannot be realized until the tradeoffs and synergies can be formulated. Integration of formal verification with empirical testing is also difficult because the idealized view of mathematical verification providing a correctness proof with total coverage is unrealistic and does not reflect the limitations imposed by computational complexity of mathematical techniques. This paper first describes a framework based on software reliability and formalized fault models for a theory of software design fault detection - and hence the utility of various tools for debugging. It then describes a utility model for integrating mathematical and empirical techniques with respect to fault detection and coverage analysis. It then considers the optimal combination of black-box testing, white-box (structural) testing, and formal methods in V&V of a software system. Using case studies from NASA software systems, it then demonstrates how this utility model can be used in practice.

Lowry, Michael↗
NASA NTRS2021

Verifying Mars 2020 Sampling and Caching Robotic Functions with Position Budgeting Process and Tool

The Mars 2020 Perseverance Rover was launched on July 30th, 2020 with one of the most complex robotic systems ever implemented on an interplanetary mission. Much of this robotic complexity resides in the Rover Sampling and Caching Subsystem (SCS) to enable collection of Martian samples for eventual return to earth and preparation of surfaces for close-up surface science observations. Two robotic arms are used by SCS: the large Robotic Arm (RA) positions the coring drill and science instruments mounted to the Turret for surface interactions, and the smaller Sample Handling Assembly (SHA) manipulates Sample Tube Assemblies (STA) within the Adaptive Caching Assembly (ACA) to prepare them for sample collection, processing, and hermetic sealing. The robotic arms interact with the Martian surface and other SCS components in many ways and in a variety of configurations, with positioning accuracy requirements ranging from tens of millimeters for some surface interactions down to sub-millimeter accuracy for some ACA interactions.This paper describes the process and tool used to calculate the SCS robotic interaction positioning budgets and verify the as-built hardware when delivered. To ensure that these robotic systems are able to perform their tasks, each robotic interaction with another element was broken down into its composite functions. To calculate a positioning budget margin for each function, an allowable was defined and then compared to the list of error sources that contribute to misalignment. Across the subsystem, over 250 functions were identified to be assessed, with almost 500 error sources feeding into their budget calculations. In addition to using as-built values in the budgets for SCS Verification and Validation (V&V) after the hardware was complete, these budgets were populated with design data during the design phase to identify areas of concern and guide hardware design to ensure positive position budget margins.Because of the size of the SCS team that had inputs to the positioning budgets and the sheer number of items in the budgets that needed to be created, updated, and verified, having a tool that would allow for simultaneous access and robust data integrity and processing was imperative. To accomplish this, a web-based MySQL database was created that allowed users to create function position budgets, link individual errors and allowables to them, and view function position budget margin reports. Each error and allowable records data for lateral, normal, angular, and clocking errors, with the ability to add as-built data for up to four different hardware builds. Margin reports can be generated for either design values alone or replacing design data with as-built data when available. These as-built reports are used for final verification of the SCS positioning requirements. Ultimately, the SCS positioning budget process and database tool led to successful interactions during test and an SCS robotic system that is ready to perform sample acquisition and caching on the surface of Mars.

Williams, Jeffrey↗
NASA NTRS2004

A Survey of Formal Methods for Intelligent Swarms

Swarms of intelligent autonomous spacecraft, involving complex behaviors and interactions, are being proposed for future space exploration missions. Such missions provide greater flexibility and offer the possibility of gathering more science data than traditional single spacecraft missions. The emergent properties of swarms make these missions powerful, but simultaneously far more difficult to design, and to assure that the proper behaviors will emerge. These missions are also considerably more complex than previous types of missions, and NASA, like other organizations, has little experience in developing or in verifying and validating these types of missions. A significant challenge when verifying and validating swarms of intelligent interacting agents is how to determine that the possible exponential interactions and emergent behaviors are producing the desired results. Assuring correct behavior and interactions of swarms will be critical to mission success. The Autonomous Nano Technology Swarm (ANTS) mission is an example of one of the swarm types of missions NASA is considering. The ANTS mission will use a swarm of picospacecraft that will fly from Earth orbit to the Asteroid Belt. Using an insect colony analogy, ANTS will be composed of specialized workers for asteroid exploration. Exploration would consist of cataloguing the mass, density, morphology, and chemical composition of the asteroids, including any anomalous concentrations of specific minerals. To perform this task, ANTS would carry miniaturized instruments, such as imagers, spectrometers, and detectors. Since ANTS and other similar missions are going to consist of autonomous spacecraft that may be out of contact with the earth for extended periods of time, and have low bandwidths due to weight constraints, it will be difficult to observe improper behavior and to correct any errors after launch. Providing V&V (verification and validation) for this type of mission is new to NASA, and represents the cutting edge in system correctness, and requires higher levels of assurance than other (traditional) missions that use a single or small number of spacecraft that are deterministic in nature and have near continuous communication access. One of the highest possible levels of assurance comes from the application of formal methods. Formal methods are mathematics-based tools and techniques for specifying and verifying (software and hardware) systems. They are particularly useful for specifying complex parallel systems, such as exemplified by the ANTS mission, where the entire system is difficult for a single person to fully understand, a problem that is multiplied with multiple developers. Once written, a formal specification can be used to prove properties of a system (e.g., the underlying system will go from one state to another or not into a specific state) and check for particular types of errors (e.g., race or livelock conditions). A formal specification can also be used as input to a model checker for further validation. This report gives the results of a survey of formal methods techniques for verification and validation of space missions that use swarm technology. Multiple formal methods were evaluated to determine their effectiveness in modeling and assuring the behavior of swarms of spacecraft using the ANTS mission as an example system. This report is the first result of the project to determine formal approaches that are promising for formally specifying swarm-based systems. From this survey, the most promising approaches were selected and are discussed relative to their possible application to the ANTS mission. Future work will include the application of an integrated approach, based on the selected approaches identified in this report, to the formal specification of the ANTS mission.

Truszkowski, Walt↗
NASA NTRS2018

Methodologies for Verification and Validation of Space Launch System (SLS) Structural Dynamic Models

Responses to challenges associated with verification and validation (V&V) of Space Launch System (SLS) structural dynamics models are presented in this paper. Four methodologies addressing specific requirements for V&V are discussed. (1) Residual Mode Augmentation (RMA), which has gained acceptance by various principals in the NASA community, defines efficient and accurate FEM modal sensitivity models that are useful in test-analysis correlation and reconciliation and parametric uncertainty studies. (2) Modified Guyan Reduction (MGR) and Harmonic Reduction (HR, introduced in 1976), developed to remedy difficulties encountered with the widely used Classical Guyan Reduction (CGR) method, are presented. MGR and HR are particularly relevant for estimation of "body dominant" target modes of shell-type SLS assemblies that have numerous "body", "breathing" and local component constituents. Realities associated with configuration features and "imperfections" cause "body" and "breathing" mode characteristics to mix resulting in a lack of clarity in the understanding and correlation of FEM- and test-derived modal data. (3) Mode Consolidation (MC) is a newly introduced procedure designed to effectively "de-feature" FEM and experimental modes of detailed structural shell assemblies for unambiguous estimation of "body" dominant target modes. Finally, (4) Experimental Mode Verification (EMV) is a procedure that addresses ambiguities associated with experimental modal analysis of complex structural systems. Specifically, EMV directly separates well-defined modal data from spurious and poorly excited modal data employing newly introduced graphical and coherence metrics.

Coppolino, Robert N.↗
NASA NTRS2022

Europa Clipper Payload Verification and Validation: Avionics-Instrument Interface Test Campaign

NASA's Europa Clipper mission will investigate Jupiter's icy moon Europa using a payload suite consisting of nine instruments to address a range of scientific objectives concerning Europa's habitability. As the project proceeds past its Critical Design Review, confidence is being built in the system's ability to achieve mission objectives through the implementation of a rigorous payload verification and validation (V&V) program. As part of this payload V&V program, instrument box-level testing was performed by the payload team to verify select instrument-avionics interface requirements. This testing was performed at JPL using the avionics testbed's Bulk Data Storage Emulator (BDSEM) with visiting instrument Test Models. This paper summarizes the Data Link test campaign involving roughly four days of functional testing per instrument, including planning, testing methods, types of issues found, and the requirement closure process. Detail is also provided on the development, deployment, and validation of a standardized analysis tool used in data reviews. This testing verified requirements related to commanding rates, loss of link, packet format, clock counters, loopback test capability, and SpaceWire jitter and skew margins. Additional risk reduction testing of basic commanding, counter behavior, science data collection and transfer, and interface swapping was also performed. Because the BDSEM venue was not originally designed to be a run for record venue, the process of characterizing venue fidelity and establishing suitability for requirement closure using data collected in this venue will also be addressed.In order to close requirements, an extensible tool was developed to post-process instrument command and telemetry data from their original binary to a human-readable format and give visibility to errors detected within the data, such as packets with Cyclic Redundancy Check errors. This tool, called payload-packet-parser, is a Python 3.9 command line tool built using a variety of open-source Python libraries. Payload-packet-parser was designed to support parsing command and telemetry packets for all Europa Clipper instruments and additional analysis tools were developed for verification of specific information interface requirements. This test campaign, including post-processing using a single parsing and verification toolset, allowed for early interface testing, alleviating testing burdens on instrument teams and buying down risk on the instrument-avionics interface by finding hardware and software issues and idiosyncrasies prior to integration with system test venues. Over twenty issues were discovered across the payload, resulting in software updates and instrument rework well in advance of any system impacts. This paper concludes with an assessment of benefits and costs of this type of testing and lessons learned.

Montanez, Leticia↗
NASA NTRS2007

Full Life-Cycle Defect Management Assessment: Initial Inspection Data Collection Results and Research Questions for Further Study

It is often the case in software projects that when schedule and budget resources are limited, the Verification and Validation (V&V) activities suffer. Fewer V&V activities can be afforded and moreover, short-term challenges can result in V&V activities being scaled back or dropped altogether. As a result, too often the default solution is to save activities for improving software quality until too late in the life-cycle, relying on late-term code inspections followed by thorough testing activities to reduce defect counts to acceptable levels. As many project managers realize, however, this is a resource-intensive way of achieving the required quality for software. The Full Life-cycle Defect Management Assessment Initiative, funded by NASA s Office of Safety and Mission Assurance under the Software Assurance Research Program, aims to address these problems by: Improving the effectiveness of early life-cycle V&V activities to make their benefits more attractive to team leads. Specifically, we focus on software inspection, a proven method that can be applied to any software work product, long before executable code has been developed; Better communicating this effectiveness to software development teams, along with suggestions for parameters to improve in the future to increase effectiveness; Analyzing the impact of early life-cycle V&V on the effectiveness and cost required for late life-cycle V&V activities, such as testing, in order to make the tradeoffs more apparent. This white paper reports on an initial milestone in this work, the development of a preliminary model of inspection effectiveness across multiple NASA Centers. This model contributes toward reaching our project goals by: Allowing an examination of inspection parameters, across different types of projects and different work products, for an analysis of factors that impact defect detection effectiveness. Allowing a comparison of this NASA-specific model to existing recommendations in the literature regarding how to plan effective inspections. Forming a baseline model which can be extended to incorporate factors describing: the numbers and types of defects that are missed by inspections; how such defects flow downstream through software development phases; how effectively they can be caught by testing activities in the late stages of development. The model has been implemented in a prototype web-enabled decision-support tool which allows developers to enter their inspection data and receive feedback based on a comparison against the model. The tool also allows users to access reusable materials (such as checklists) from projects included in the baseline. Both the tool itself and the model underlying it will continue to be extended throughout the remainder of this initiative. As results of analyzing inspection effectiveness for defect containment are determined, they can be shared via the tool and also via updates to existing training courses on metrics and software inspections. Moreover, the tool will help satisfy key CMMI requirements for the NASA Centers, as it will enable NASA to take a global view across peer review results for various types of projects to identify systemic problems. This analysis can result in continuous improvements to the approach to verification.

Shull, Forrest↗
NASA NTRS1987

A formal approach to validation and verification for knowledge-based control systems

As control systems become more complex in response to desires for greater system flexibility, performance and reliability, the promise is held out that artificial intelligence might provide the means for building such systems. An obstacle to the use of symbolic processing constructs in this domain is the need for verification and validation (V and V) of the systems. Techniques currently in use do not seem appropriate for knowledge-based software. An outline of a formal approach to V and V for knowledge-based control systems is presented.

Castore, Glen↗
NASA NTRS1988

Knowledge based system verification and validation as related to automation of space station subsystems: Rationale for a knowledge based system lifecycle

The role of verification and validation (V and V) in software has been to support and strengthen the software lifecycle and to ensure that the resultant code meets the standards of the requirements documents. Knowledge Based System (KBS) V and V should serve the same role, but the KBS lifecycle is ill-defined. The rationale of a simple form of the KBS lifecycle is presented, including accommodation to certain critical KBS differences from software development.

Richardson, Keith↗
NASA NTRS1988

Workshop on Verification and Validation of Knowledge Based Systems. I - Overview and meeting summary

An overview is presented of a workshop held on May 14-15, 1987. Presentations included overviews of knowledge-based system (KBS) fundamentals, introductions to verification and validation (V&V) methods currently being used, and theoretical V&V work. The findings of three working groups are considered, involving: (1) KBS V&V feasibility within current NASA software constraints, (2) V&V methods suited to KBS constraints, and future KBS V&V research goals.

Richardson, Keith↗
NASA NTRS1988

Knowledge based system verification and validation as related to automation of Space Station subsystems - Rationale for a knowledge based system lifecycle

The role of verification and validation (V and V) in software has been to support and strengthen the software lifecycle and to ensure that the resultant code meets the standards of the requirements documents. Knowledge-based system (KBS) V and V should serve the same role, but the KBS lifecycle is ill-defined. The rationale of a simple form of the KBS lifecycle is presented, including accommodation to certain critical KBS differences from software development.

Richardson, Keith↗
NASA NTRS1987

An expert system development methodology which supports verification and validation

Expert systems have demonstrated commercial viability in a wide range of applications, but still face some obstacles to widespread use. A major stumbling block is the lack of well defined verification and validation (V and V) techniques. The primary difficulty with expert system V and V is the use of development methodologies which do not support V and V. As with conventional code, the key to effective V and V is the development methodology. An expert system development methodology is described which is based upon a panel review approach, that allows input from all parties concerned with the expert system.

Culbert, Chris↗
NASA NTRS1990

Space station advanced automation

In the development of a safe, productive and maintainable space station, Automation and Robotics (A and R) has been identified as an enabling technology which will allow efficient operation at a reasonable cost. The Space Station Freedom's (SSF) systems are very complex, and interdependent. The usage of Advanced Automation (AA) will help restructure, and integrate system status so that station and ground personnel can operate more efficiently. To use AA technology for the augmentation of system management functions requires a development model which consists of well defined phases of: evaluation, development, integration, and maintenance. The evaluation phase will consider system management functions against traditional solutions, implementation techniques and requirements; the end result of this phase should be a well developed concept along with a feasibility analysis. In the development phase the AA system will be developed in accordance with a traditional Life Cycle Model (LCM) modified for Knowledge Based System (KBS) applications. A way by which both knowledge bases and reasoning techniques can be reused to control costs is explained. During the integration phase the KBS software must be integrated with conventional software, and verified and validated. The Verification and Validation (V and V) techniques applicable to these KBS are based on the ideas of consistency, minimal competency, and graph theory. The maintenance phase will be aided by having well designed and documented KBS software.

Woods, Donald↗
NASA NTRS1990

Expert system verification and validation survey. Delivery 2: Survey results

The purpose is to determine the state-of-the-practice in Verification and Validation (V and V) of Expert Systems (ESs) on current NASA and industry applications. This is the first task of the series which has the ultimate purpose of ensuring that adequate ES V and V tools and techniques are available for Space Station Knowledge Based Systems development. The strategy for determining the state-of-the-practice is to check how well each of the known ES V and V issues are being addressed and to what extent they have impacted the development of ESs.

Source record↗