SEARCH · Search NASA

Results for “Verification and Validation (V&V)”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15

All sources NASA NTRS DOE OSTI

NASA NTRS2011

Ares I-X Range Safety Simulation and Analysis IV and V

NASA s Ares I-X vehicle launched on a suborbital test flight from the Eastern Range in Florida on October 28, 2009. NASA generated a Range Safety (RS) product data package to meet the RS trajectory data requirements defined in the Air Force Space Command Manual (AFSPCMAN) 91-710. Some products included were a nominal ascent trajectory, ascent flight envelopes, and malfunction turn data. These products are used by the Air Force s 45th Space Wing (45SW) to ensure public safety and to make flight termination decisions on launch day. Due to the criticality of the RS data, an independent validation and verification (IV&V) effort was undertaken to accompany the data generation analyses to ensure utmost data quality and correct adherence to requirements. As a result of the IV&V efforts, the RS product package was delivered with confidence that two independent organizations using separate simulation software generated data to meet the range requirements and yielded similar results. This document captures the Ares I-X RS product IV&V analysis, including the methodology used to verify inputs, simulation, and output data for certain RS products. Additionally a discussion of lessons learned is presented to capture advantages and disadvantages to the IV&V processes used.

Merry, Carl M.↗

DOE OSTI2021

Evaluating Symmetry in 3-dimensional Simulation Data

The increasing use of three-dimensional simulations has led to an increased need for effective verification and validation, or V&V. Unfortunately, the high computational cost associated with three-dimensional simulations makes many standard V&V techniques cost-prohibitive. Given these circumstances, we must pay close attention to V&V metrics that can be evaluated using only a single simulation. One of the simplest of these metrics is the evaluation of physical symmetries. This report details a method of computing the deviation from symmetry in a three-dimensional dataset with an arbitrarily distributed mesh. Examples are presented for spherical and axisymmetric systems using the Paraview post-processing analysis workflow.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Efforts to enhance reproducibility in a human performance research project

Background: Ensuring the validity of results from funded programs is a critical concern for agencies that sponsor biological research. In recent years, the open science movement has sought to promote reproducibility by encouraging sharing not only of finished manuscripts but also of data and code supporting their findings. While these innovations have lent support to third-party efforts to replicate calculations underlying key results in the scientific literature, fields of inquiry where privacy considerations or other sensitivities preclude the broad distribution of raw data or analysis may require a more targeted approach to promote the quality of research output. Methods: We describe efforts oriented toward this goal that were implemented in one human performance research program, Measuring Biological Aptitude, organized by the Defense Advanced Research Project Agency's Biological Technologies Office. Our team implemented a four-pronged independent verification and validation (IV&V) strategy including 1) a centralized data storage and exchange platform, 2) quality assurance and quality control (QA/QC) of data collection, 3) test and evaluation of performer models, and 4) an archival software and data repository. Results: Our IV&V plan was carried out with assistance from both the funding agency and participating teams of researchers. QA/QC of data acquisition aided in process improvement and the flagging of experimental errors. Holdout validation set tests provided an independent gauge of model performance. Conclusions: In circumstances that do not support a fully open approach to scientific criticism, standing up independent teams to cross-check and validate the results generated by primary investigators can be an important tool to promote reproducibility of results.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2022

Emulation Modeling for Development of Cyber-Defense Capabilities for Satellite Systems

The objective of this project was to develop a novel capability to generate synthetic data sets for the purpose of training Machine Learning (ML) algorithms for the detection of malicious activities on satellite systems. The approach experimented with was to a) generate sparse data sets using emulation modeling and b) enlarge the sparse data using Generative Adversarial Networks (GANs). We based our emulation modeling on the Open Source NASA Operational Simulator for Small Satellites (NOS3) developed by the Katherine Johnson Independent Verification and Validation (IV&V) program in West Virginia. Significant new capabilities on NOS3 had to be developed for our data set generation needs. To expand these data sets for the purpose of training ML, we experimented with a) Extreme Learning Machines (ELMs) and b) Wasserstein-GANs (WGAN-GP).

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

NASA NTRS1985

A practical experience with independent verification and validation

One approach to reducing software cost and increasing reliability is the use of an independent verification and validation (IV & V) methodology. The Software Engineering Laboratory (SEL) applied the IV & V methodology to two medium-size flight dynamics software development projects. Then, to measure the effectiveness of the IV & V approach, the SEL compared these two projects with two similar past projects, using measures like productivity, reliability, and maintain ablilty. Results indicated that the use of the IV & V methodology did not help the overall process nor improve the product in these cases.

Page, Gerald↗

NASA NTRS1992

Independent verification and validation for Space Shuttle flight software

The Committee for Review of Oversight Mechanisms for Space Shuttle Software was asked by the National Aeronautics and Space Administration's (NASA) Office of Space Flight to determine the need to continue independent verification and validation (IV&V) for Space Shuttle flight software. The Committee found that the current IV&V process is necessary to maintain NASA's stringent safety and quality requirements for man-rated vehicles. Therefore, the Committee does not support NASA's plan to eliminate funding for the IV&V effort in fiscal year 1993. The Committee believes that the Space Shuttle software development process is not adequate without IV&V and that elimination of IV&V as currently practiced will adversely affect the overall quality and safety of the software, both now and in the future. Furthermore, the Committee was told that no organization within NASA has the expertise or the manpower to replace the current IV&V function in a timely fashion, nor will building this expertise elsewhere necessarily reduce cost. Thus, the Committee does not recommend moving IV&V functions to other organizations within NASA unless the current IV&V is maintained for as long as it takes to build comparable expertise in the replacing organization.

Source record↗

NASA NTRS1996

Independent Verification and Validation of Complex User Interfaces: A Human Factors Approach

The Usability Testing and Analysis Facility (UTAF) at the NASA Johnson Space Center has identified and evaluated a potential automated software interface inspection tool capable of assessing the degree to which space-related critical and high-risk software system user interfaces meet objective human factors standards across each NASA program and project. Testing consisted of two distinct phases. Phase 1 compared analysis times and similarity of results for the automated tool and for human-computer interface (HCI) experts. In Phase 2, HCI experts critiqued the prototype tool's user interface. Based on this evaluation, it appears that a more fully developed version of the tool will be a promising complement to a human factors-oriented independent verification and validation (IV&V) process.

Whitmore, Mihriban↗

NASA NTRS1997

Formal Methods for Verification and Validation of Partial Specifications: A Case Study

This paper describes our work exploring the suitability of formal specification methods for independent verification and validation (IV&V) of software specifications for large, safety critical systems. An IV&V contractor often has to perform rapid analysis on incomplete specifications, with no control over how those specifications are represented. Lightweight formal methods show significant promise in this context, as they offer a way of uncovering major errors, without the burden of full proofs of correctness. We describe a case study of the use of partial formal models for V&V of the requirements for Fault Detection Isolation and Recovery on the space station. We conclude that the insights gained from formalizing a specification are valuable, and it is the process of formalization, rather than the end product that is important. It was only necessary to build enough of the formal model to test the properties in which we were interested. Maintenance of fidelity between multiple representations of the same requirements (as they evolve) is still a problem, and deserves further study.

Easterbrook, Steve↗

NASA NTRS1997

Applying Independent Verification and Validation to Automatic Test Equipment

This paper describes a general overview of applying Independent Verification and Validation (IV&V) to Automatic Test Equipment (ATE). The overview is not inclusive of all IV&V activities that can occur or of all development and maintenance items that can be validated and verified, during the IV&V process. A sampling of possible IV&V activities that can occur within each phase of the ATE life cycle are described.

Calhoun, Cynthia C.↗

NASA NTRS1997

A Case Study of 4 & 5 Cost Effectiveness

This paper looks at the Independent Verification and Validation (IV&V) of NASA's Space Shuttle Day of Launch I-Load Update (DoLILU) project. IV&V is defined. The system's development life cycle is explained. Data collection and analysis are described. DoLILU Issue Tracking Reports (DITRs) authored by IV&V personnel are analyzed to determine the effectiveness of IV&V in finding errors before the code, testing, and integration phase of the software development life cycle. The study's findings are reported along with the limitations of the study and planned future research.

Neal, Ralph D.↗

NASA NTRS2003

Experimental Evaluation of Verification and Validation Tools on Martian Rover Software

We report on a study to determine the maturity of different verification and validation technologies (V&V) on a representative example of NASA flight software. The study consisted of a controlled experiment where three technologies (static analysis, runtime analysis and model checking) were compared to traditional testing with respect to their ability to find seeded errors in a prototype Mars Rover. What makes this study unique is that it is the first (to the best of our knowledge) to do a controlled experiment to compare formal methods based tools to testing on a realistic industrial-size example where the emphasis was on collecting as much data on the performance of the tools and the participants as possible. The paper includes a description of the Rover code that was analyzed, the tools used as well as a detailed description of the experimental setup and the results. Due to the complexity of setting up the experiment, our results can not be generalized, but we believe it can still serve as a valuable point of reference for future studies of this kind. It did confirm the belief we had that advanced tools can outperform testing when trying to locate concurrency errors. Furthermore the results of the experiment inspired a novel framework for testing the next generation of the Rover.

Brat, Guillaume↗

NASA NTRS2003

Reducing software security risk through an integrated approach research initiative model based verification of the Secure Socket Layer (SSL) Protocol

This document discusses the verification of the Secure Socket Layer (SSL) communication protocol as a demonstration of the Model Based Verification (MBV) portion of the verification instrument set being developed under the Reducing Software Security Risk (RSSR) Trough an Integrated Approach research initiative. Code Q of the National Aeronautics and Space Administration (NASA) funds this project. The NASA Goddard Independent Verification and Validation (IV&V) facility manages this research program at the NASA agency level and the Assurance Technology Program Office (ATPO) manages the research locally at the Jet Propulsion Laboratory (California institute of Technology) where the research is being carried out.

software security↗

NASA NTRS2006

A Predictive Approach to Eliminating Errors in Software Code

NASA s Metrics Data Program Data Repository is a database that stores problem, product, and metrics data. The primary goal of this data repository is to provide project data to the software community. In doing so, the Metrics Data Program collects artifacts from a large NASA dataset, generates metrics on the artifacts, and then generates reports that are made available to the public at no cost. The data that are made available to general users have been sanitized and authorized for publication through the Metrics Data Program Web site by officials representing the projects from which the data originated. The data repository is operated by NASA s Independent Verification and Validation (IV&V) Facility, which is located in Fairmont, West Virginia, a high-tech hub for emerging innovation in the Mountain State. The IV&V Facility was founded in 1993, under the NASA Office of Safety and Mission Assurance, as a direct result of recommendations made by the National Research Council and the Report of the Presidential Commission on the Space Shuttle Challenger Accident. Today, under the direction of Goddard Space Flight Center, the IV&V Facility continues its mission to provide the highest achievable levels of safety and cost-effectiveness for mission-critical software. By extending its data to public users, the facility has helped improve the safety, reliability, and quality of complex software systems throughout private industry and other government agencies. Integrated Software Metrics, Inc., is one of the organizations that has benefited from studying the metrics data. As a result, the company has evolved into a leading developer of innovative software-error prediction tools that help organizations deliver better software, on time and on budget.

Source record↗

NASA NTRS2009

IT Project Success w\7120 and 7123 NPRs to Achieve Project Success

This slide presentation reviews management techniques to assure information technology development project success. Details include the work products, the work breakdown structure (WBS), system integration, verification and validation (IV&V), and deployment and operations. An example, the NASA Consolidated Active Directory (NCAD), is reviewed.

Walley, Tina L.↗

NASA NTRS2010

Using Web 2.0 Techniques in NASA's Ares Engineering Operations Network (AEON) Environment - First Impressions

The Mission Operations Laboratory (MOL) at Marshall Space Flight Center (MSFC) is responsible for Engineering Support capability for NASA s Ares rocket development and operations. In pursuit of this, MOL is building the Ares Engineering and Operations Network (AEON), a web-based portal to support and simplify two critical activities: Access and analyze Ares manufacturing, test, and flight performance data, with access to Shuttle data for comparison Establish and maintain collaborative communities within the Ares teams/subteams and with other projects, e.g., Space Shuttle, International Space Station (ISS). AEON seeks to provide a seamless interface to a) locally developed engineering applications and b) a Commercial-Off-The-Shelf (COTS) collaborative environment that includes Web 2.0 capabilities, e.g., blogging, wikis, and social networking. This paper discusses how Web 2.0 might be applied to the typically conservative engineering support arena, based on feedback from Integration, Verification, and Validation (IV&V) testing and on searching for their use in similar environments.

Scott, David W.↗

NASA NTRS2016

Technical Reference Suite Addressing Challenges of Providing Assurance for Fault Management Architectural Design

Research into complexities of software systems Fault Management (FM) and how architectural design decisions affect safety, preservation of assets, and maintenance of desired system functionality has coalesced into a technical reference (TR) suite that advances the provision of safety and mission assurance. The NASA Independent Verification and Validation (IV&V) Program, with Software Assurance Research Program support, extracted FM architectures across the IV&V portfolio to evaluate robustness, assess visibility for validation and test, and define software assurance methods applied to the architectures and designs. This investigation spanned IV&V projects with seven different primary developers, a wide range of sizes and complexities, and encompassed Deep Space Robotic, Human Spaceflight, and Earth Orbiter mission FM architectures. The initiative continues with an expansion of the TR suite to include Launch Vehicles, adding the benefit of investigating differences intrinsic to model-based FM architectures and insight into complexities of FM within an Agile software development environment, in order to improve awareness of how nontraditional processes affect FM architectural design and system health management. The identification of particular FM architectures, visibility, and associated IV&V techniques provides a TR suite that enables greater assurance that critical software systems will adequately protect against faults and respond to adverse conditions. Additionally, the role FM has with regard to strengthened security requirements, with potential to advance overall asset protection of flight software systems, is being addressed with the development of an adverse conditions database encompassing flight software vulnerabilities. Capitalizing on the established framework, this TR suite provides assurance capability for a variety of FM architectures and varied development approaches. Research results are being disseminated across NASA, other agencies, and the software community. This paper discusses the findings and TR suite informing the FM domain in best practices for FM architectural design, visibility observations, and methods employed for IV&V and mission assurance.

Fitz, Rhonda↗

NASA NTRS2019

Agile Approach to Assuring the Safety-Critical Embedded Software for NASA's Orion Spacecraft

Human-rated missions like those in NASA's Orion Program continue to grow in complexity. The role of software in achieving ambitious mission objectives has expanded dramatically in the last few decades. Assuring the safety and performance of the embedded flight software is quickly growing beyond the reach of traditional methods and resource levels. The methods used to build these software-dominant systems evolve in an on-going attempt to keep pace with the scope of our ambitions. Agile software development is now commonplace. The long timelines and large batches of work associated with traditional methods are being replaced by rapid delivery of small increments _ as system capabilities are realized in waves. Assurance of these critical software capabilities must therefore conquer an ever-expanding frontier of challenges, and do so with an approach matched to the evolving development methods. This paper recounts the journey of the Orion Independent Verification and Validation (IV&V) team as we addressed this dynamic environment. Widening our aperture to encompass a dramatically larger mission scope, while adjusting our cadence to synchronize with the rapid pace of agile software development, a new approach to IV&V is emerging. This approach is characterized by a sharper focus on mission capabilities, matched with a method to dynamically _follow the risk' as the IV&V team delivers more compelling assurance data in waves. Traditional methods prevalent in IV&V tend to scope the work using artifacts of the development process as they evolve from preliminary to final versions, and the pace of delivery was synchronized with the development timelines prevalent in the waterfall lifecycle. That more static approach is out of phase with the demands of the new environment. Scoping work according to the critical capabilities of the system (rather than artifacts of development) and synchronizing with the rapid pace of agile development, we are moving toward more effective parity with the demands of the environment. We explain the concrete steps we took, the principles that motivated our choices, and the results we have achieved to date.

Capability based assurance↗

NASA NTRS2019

Agile Approach to Assuring the Safety-Critical Embedded Software for NASA's Orion Spacecraft

Human-rated missions like NASA's Exploration Mission - 1 (EM-1) and the Orion Multi-Purpose Crew Vehicle are becoming exceedingly complex in terms of software's contribution to achieving mission objectives. The increasing complexity and inherent safety critical nature of the embedded flight software imposes a unique resource challenge to assurance providers responsible for affirming that the mission is going to fly safely. Another challenge NASA and other Government agencies are facing is that more and more software is being developed using an agile development methodology, which is divergent from the typical waterfall, iterative, and incremental development methodologies assurance providers generally observe in the development of safety-critical embedded software. Orion Independent Verification and Validation (IV&V) has addressed these challenges by providing focused assurance results of critical mission capabilities prioritized by a dynamic assessment of risk level. Prior to this approach, Orion IV&V evaluated areas of risk in much broader, and more static, terms. Due to the agile software development life cycle that Orion follows, IV&V findings were often reported months out of phase with the developer thereby imposing increased rework costs. As a result of evolving the approach to adding assurance on Orion, IV&V is able to incrementally deliver high-priority assurance conclusions and more impactful issues more in phase with the developer activities, thereby increasing the value of the findings to the project. The agile IV&V approach employed by the Orion IV&V team strives to achieve a cadence of delivery that matches the pace of development. This agile approach provides increased flexibility for the assurance provider to become more efficient in reporting assurance conclusions and issues. This paper and presentation will discuss the principles which drive the design of our approach, results to date, and stimulate thinking for groups looking to add assurance to software being developed using an agile methodology.

Justin Smith↗