Search NASA⌕ Search

SEARCH · Search NASA

Results for “Verification and Validation (V&V)”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Adaptive Stress Testing of Collision Avoidance Systems for Small UASs with Deep Reinforcement Learning

The next-generation Airborne Collision Avoidance System for smaller UASs (ACAS sXu) is currently being developed and tested by the Federal Aviation Administration (FAA) to provide detect-and-avoid capability for small unmanned aircraft operating beyond line-of-sight. Due to the complexity and safety-critical nature of the system, safety validation is important not only for the certification of the final system, but also for informing changes during the iterative development process. In this paper, we analyze a prototype of ACAS sXu in simulated aircraft encounters to discover scenarios of small near mid-air collisions (sNMACs), an important safety event in which two aircraft come closer than 50 feet horizontally and 15 feet vertically. Due to the size and complexity of the system as well as rarity of sNMAC events, traditional methods such as Monte Carlo testing often require informed setup and targeting to elicit failures. However, such a dependence on domain knowledge can be incompatible with the independent verification and validation (IV&V) process, the aim of which is to discover unforeseen issues. To address these challenges, we apply an accelerated validation method called adaptive stress testing (AST) to find the most likely sNMAC scenarios without reliance on system introspection. AST uses reinforcement learning to adapt the search towards the most promising areas of the search space as it progresses. We use a state-of-the-art deep reinforcement learning algorithm, proximate policy optimization, to more efficiently search the large and continuous state space. We find that this approach significantly improves the performance of AST compared to a prior approach based on Monte Carlo tree search. We perform experiments using AST to find sNMAC events under various encounter configurations, varying parameters pertaining to dynamics and coordination. Our experiments show AST to be very effective at finding sNMAC scenarios. We summarize our findings, presenting high-level categories of discovered sNMACs and specific examples of encounters in each category.

aircraft collision avoidance↗
NASA NTRSDate not supplied

Alternate Methods of Model Reduction to Avoid Dynamic Modal Truncation Error

Loads analysis is traditionally performed using dynamically reduced models, which provides the benefit to reduce run time. If the reduced model frequency cutoff is not chosen appropriately, the model will lack the dynamic content required to fully represent the response of the non-reduced model. Standard guidance for reduced model frequency content, provided in NASA-STD-5002, is to solve fixed base modes up to a minimum of 1.5x the model frequency content of interest and to employ static modal truncation methods such as residual vectors, the mode acceleration method, and the residual flexibility method to account for the truncated flexibility of the missing modes. Fixed base modes require the selection of a set of degrees of freedom to be constrained which, if not properly selected, may affect the accuracy of the reduced model by excluding some of the dynamic characteristic of the full model. In this case, the standard NASA guidance would be insufficient, but it may not be readily apparent that a portion of the reduced model response is missing. This error was encountered during an independent verification and validation (IV&V) effort, where it was observed that the resulting dynamic response was lower than the inline analysis. In this specific case, despite following the standard NASA model reduction guidelines in the selection of the frequency cutoff, the inline model still did not fully capture the necessary dynamic content. As part of the IV&V, an alternate reduction methodology was employed using an unconstrained mode acceleration method. The original model initially performed a constrained reduction to twice the frequency content of interest before doing a free-free run, employing the mode acceleration method to account for the truncated modes. In contrast in the IV&V, the unconstrained model reduced directly to the needed frequency content of the free-free run, avoiding any interactions between constraints and dynamic content. To verify the model, the original reduction methodology was used to generate a series of Hurty-Craig-Bampton reductions, each with a higher frequency cutoff than the previous. The results were shown to converge once the frequency cutoff increased past eight times the frequency content of interest. At the same frequency cutoff, the results of the Hurty-Craig-Bampton model converged with the results of the unconstrained mode acceleration model. This comparative study provided confidence that the results of the unconstrained modal acceleration reduced model were correct and that the Hurty-Craig-Bampton needed to increase its frequency cutoff to fully capture the dynamic response.

Erin Simmons↗
NASA NTRSDate not supplied

Alternate Methods of Model Reduction to Avoid Dynamic Modal Truncation Error

Loads analysis is traditionally performed using dynamically reduced models, which provides the benefit to reduce run time. If the reduced model frequency cutoff is not chosen appropriately, the model will lack the dynamic content required to fully represent the response of the non-reduced model. Standard guidance for reduced model frequency content, provided in NASA-STD-5002, is to solve fixed base modes up to a minimum of 1.5x the model frequency content of interest and to employ static modal truncation methods such as residual vectors, the mode acceleration method, and the residual flexibility method to account for the truncated flexibility of the missing modes. Fixed base modes require the selection of a set of degrees of freedom to be constrained which, if not properly selected, may affect the accuracy of the reduced model by excluding some of the dynamic characteristic of the full model. In this case, the standard NASA guidance would be insufficient, but it may not be readily apparent that a portion of the reduced model response is missing. This error was encountered during an independent verification and validation (IV&V) effort, where it was observed that the resulting dynamic response was lower than the inline analysis. In this specific case, despite following the standard NASA model reduction guidelines in the selection of the frequency cutoff, the inline model still did not fully capture the necessary dynamic content. As part of the IV&V, an alternate reduction methodology was employed using an unconstrained mode acceleration method. The original model initially performed a constrained reduction to twice the frequency content of interest before doing a free-free run, employing the mode acceleration method to account for the truncated modes. In contrast in the IV&V, the unconstrained model reduced directly to the needed frequency content of the free-free run, avoiding any interactions between constraints and dynamic content. To verify the model, the original reduction methodology was used to generate a series of Hurty-Craig-Bampton reductions, each with a higher frequency cutoff than the previous. The results were shown to converge once the frequency cutoff increased past eight times the frequency content of interest. At the same frequency cutoff, the results of the Hurty-Craig-Bampton model converged with the results of the unconstrained mode acceleration model. This comparative study provided confidence that the results of the unconstrained modal acceleration reduced model were correct and that the Hurty-Craig-Bampton needed to increase its frequency cutoff to fully capture the dynamic response.

Erin Simmons↗
DOE OSTI2022

MPACT Verification and Validation Manual (V.4.3)

As the VERA SQA plan requires, it is the responsibility of the University of Michigan (UM) and Oak Ridge National Laboratory (ORNL), as co-owners of MPACT, to ensure that verification and validation activities are performed and documented in a V&V manual with supporting publications and CASL technical reports which can be provided for reference and distribution within VERA. This document provides the current revision of the MPACT verification and validation (V&V) manual and describes the current state of MPACT V&V and updates the plans for future MPACT V&V activities. The following sections provide an overview of the V&V process used in MPACT, as well as a summary of the status of each component of V&V in the code.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

MPACT Verification and Validation Manual Version 4.4

As the VERA SQA plan requires, it is the responsibility of the University of Michigan (UM) and Oak Ridge National Laboratory (ORNL), as co-owners of MPACT, to ensure that verification and validation activities are performed and documented in a V&V manual with supporting publications and CASL technical reports which can be provided for reference and distribution within VERA. This document provides the current revision of the MPACT verification and validation (V&V) manual and describes the current state of MPACT V&V and updates the plans for future MPACT V&V activities. The following sections provide an overview of the V&V process used in MPACT, as well as a summary of the status of each component of V&V in the code.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2020

Validation of finite-element models using full-field experimental data: Levelling finite-element analysis data through a digital image correlation engine

Full-field data from digital image correlation (DIC) provide rich information for finite-element analysis (FEA) validation. However, there are several inherent inconsistencies between FEA and DIC data that must be rectified before meaningful, quantitative comparisons can be made, including strain formulations, coordinate systems, data locations, strain calculation algorithms, spatial resolutions and data filtering. As such, in this paper, we investigate two full-field validation approaches: (1) the direct interpolation approach, which addresses the first three inconsistencies by interpolating the quantity of interest from one mesh to the other, and (2) the proposed DIC-levelling approach, which addresses all six inconsistencies simultaneously by processing the FEA data through a stereo-DIC simulator to ‘level’ the FEA data to the DIC data in a regularisation sense. Synthetic ‘experimental’ DIC data were generated based on a reference FEA of an exemplar test specimen. The direct interpolation approach was applied, and significant strain errorswere computed, even though therewas no model form error, because the filtering effect of theDIC enginewas neglected. In contrast, the levelling approach provided accurate validation results, with no strain error when no model form error was present. Next, model form error was purposefully introduced via a mismatch of boundary conditions. With the direct interpolation approach, the mismatch in boundary conditions was completely obfuscated, while with the levelling approach, it was clearly observed. Finally, the ‘experimental’ DIC datawere purposefully misaligned slightly fromthe FEA data. Both validation techniques suffered from the misalignment, thus motivating continued efforts to develop a robust alignment process. In conclusion, direct interpolation is insufficient, and the proposed levelling approach is required to ensure that the FEA and the DIC data have the same spatial resolution and data filtering. Only after the FEA data have been ‘levelled’ to the DIC data can meaningful, quantitative error maps be computed.

42 ENGINEERING↗
DOE OSTI2022

MCNP6.3: A Year in Review [Slides]

This presentation discusses the past year and the many accomplishments of the MCNP6.3 team. It states that the MCNP6.3 release is imminent and that the approved, final documents have been making their way to the website. The code executables and source are already packaged up for distribution and the new installer is being finalized and tested now, for all platforms. The package will be sent to RSICC before the end of October 2022. Additionally, the presentation discusses some things to think about as MCNP6.3 is requested and/or used. For example, many of the recent efforts have been focused on making development, updates, and distribution of the code and documents more robust and streamlined. They will be revising/updating documents more frequently than ever before and they will be exploring avenues to distribute official patches to MCNP6.3. As they explain, this allows them to be more responsive to bugs and issues that are identified. In conclusion, the application of a patch to MCNP6.3 will require having the source code.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2023

S4PST: Sustainability for Programming Systems and Tools: May Workshop Report

The US Department of Energy (DOE) Exascale Computing Project (ECP) has fostered and strengthened the use of modern software engineering practices for developing applications and libraries, and this effort has resulted in the coordinated and interoperable E4S1 and xSDK2 ecosystems. Although this approach is cost-effective, it relies on robust programming systems and tools (PST) as the underlying foundation for our HPC software. At present, our primary PST stack consists of traditional high-performance computing (HPC) languages, namely Fortran, C, C++, and the popular Python language for data analysis and AI workflows. These languages support various programming frameworks and run-time abstractions that enable parallelism and concurrency across multiple node architectures and thousands of nodes through a variety of interconnect systems. However, to accommodate users’ diverse needs, certain aspects of the HPC ecosystem are delegated to vendor-specific or third-party implementations that extend beyond a particular scientific domain. This broader scope results in a multitude of specifications and variations, which leads to a complex orchestration of many-ecosystems. Unfortunately, this complexity in the ecosystem imposes additional overhead costs on consumers during the latter stages of the development cycle. In addition to the software ecosystem challenge, the upcoming conclusion of the ECP by December 2023 has raised significant concerns within the HPC programming systems community, from both the economic and social perspectives. The ECP has implemented a management structure for software development and funding decisions across all ECP participants by following a conventional hierarchical and centralized approach. However, this structure has prompted certain considerations within the community, particularly in anticipation of the Software Sustainability initiative by the DOE’s Advanced Scientific Computing Research Program (ASCR). For the success of this new initiative, it is of utmost importance to secure consistent funding and foster close engagement with researchers and core developers of existing programming-system products. This collaboration is vital to maintaining the critical capabilities of the current software during the transition phase while proactively adapting to future technology and workforce trends. The community recognizes the significance of adapting to emerging trends and is aware of the inherent fragility of the HPC software ecosystem, particularly in relation to programming systems that cater to all users. The ability to adapt and evolve is essential to staying relevant and effectively addressing these technical, economic, and social challenges. The S4PST team, which represents one of the six ASCR Software Sustainability seedling projects, is dedicated to tackling these challenges through community-based approaches that go beyond the scope of the DOE. This involves collaboration between national laboratories with academia, non-DOE institutions, hardware and system vendors, and international partners. By fostering these partnerships, we aim to create a robust and sustainable HPC software ecosystem that can effectively meet the needs of the community. This new community effort, driven by the eight DOE labs, will take on the responsibility of guiding funding decisions for programming-systems development and maintenance with transparency and consistency across all decisions. Additionally, the team will offer common technical services to the programming systems community, irrespective of their funding situations, and facilitate community-wide incubation to proactively nurture the software ecosystem. By actively engaging with stakeholders and employing a collaborative approach, we can collectively shape the future of programming systems and ensure a robust and thriving HPC software landscape. On May 11–12, 2023, the S4PST team conducted its inaugural kick-off workshop at the Innovative Computing Laboratory (ICL) in the University of Tennessee, Knoxville, hosted by Hartwig Anzt. The workshop encompassed various sessions dedicated to presentations and discussions, with the aim of comprehending the team members’ perspectives on the vision of software sustainability. Additionally, the workshop aimed to identify the technical, economic, and social requirements for sustaining the programming-systems community in the field of HPC. This report provides a summary of the S4PST effort by highlighting five major thrust areas discussed during the workshop: (i) community, (ii) technical support, (iii) training and diversity, (iv) verification, validation and correctness, and (v) emerging technologies. It also encompasses an overview of the presentations and discussions held throughout the event, our views and potential synergies with other seedling efforts, along with the outcomes and key takeaways from our initial discussions.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1999

Model Checking Verification and Validation at JPL and the NASA Fairmont IV and V Facility

We show how a technology transfer effort was carried out. The successful use of model checking on a pilot JPL flight project demonstrates the usefulness and the efficacy of the approach. The pilot project was used to model a complex spacecraft controller. Software design and implementation validation were carried out successfully. To suggest future applications we also show how the implementation validation step can be automated. The effort was followed by the formal introduction of the modeling technique as a part of the JPL Quality Assurance process.

Schneider, Frank↗
NASA NTRS2018

Dynamic Power Convertor Development for Radioisotope Power Systems at NASA Glenn Research Center

The Thermal Energy Conversion Branch at NASA Glenn Research Center is supporting the development of high-efficiency power convertors for use in radioisotope power systems (RPS). Significant progress was made toward such a system that utilized Stirling conversion during the 2001 to 2015 timeframe. Flight development of the Advanced Stirling Radioisotope Generator (ASRG) was canceled in 2013 by the U.S. Department of Energy (DOE) and NASA Headquarters primarily due to budget constraints, and the Advanced Stirling Convertor (ASC) technology contract was subsequently concluded in 2015. A new chapter of technology development has recently been initiated by the NASA RPS Program. This effort is considering all dynamic power convertor options, such as Stirling and Brayton cycles. Four convertor development contracts supporting this effort were awarded in 2017. The awarded contracts include two free-piston Stirling, one thermoacoustic Stirling, and one turbo-Brayton design. The technology development contracts each consist of up to three phases: design, fabrication, and test. As of May 2018, all contracts have completed the design phase, and each underwent a design review with an independent review board. Three of the contracts are planned to execute the phase 2 option for fabrication. Convertors manifesting from these development efforts will then undergo independent validation and verification (IV and V) at NASA facilities, which will consist of convertor performance and RPS viability demonstrations. Example tests include launch vibration simulation, performance mapping over the environmental temperature range, and static acceleration exposure. In parallel with this renewed development effort, Glenn is still demonstrating free-piston Stirling convertor technology using assets from previous projects. The Stirling Research Laboratory (SRL) is still operating several convertors from previous development projects, which have similarities and relevance to current contract designs. Four of these are flexure-bearing based and another six are gas bearing based. One of the flexure-bearing convertors has accumulated over 110,000 h of operation and holds the current record for maintenance-free heat-engine run time. Another flexure-bearing convertor was recently manually shut down after 105,620 h of operation, then disassembled and inspected. This inspection produced a wealth of information about the effects of this amount of run time on the technology's components. One of the engineering unit flexure-bearing convertors recently underwent a launch simulation vibration test, a static acceleration exposure up to 20 g, and was then placed on extended operation. Among the gas-bearing convertors, the longest running unit has accumulated over 70,000 h of operation. Four high-fidelity gas-bearing convertors from the ASRG project are still operating continuously for which the longest run time has reached 28,000 h.

Oriti, Salvatore M.↗
NASA NTRSDate not supplied

Software-Only Key Management and Cryptography (KMC) for Spacecraft Command Encryption

The Jet Propulsion Laboratory (JPL) Advanced Multi-Mission Operations System (AMMOS) in partnership with NASA’s Katherine Johnson Independent Verification and Validation (NASA IV&V) Jon McBride Software Testing and Research (JSTAR) laboratory have jointly developed a command encryption capability that provides a software-only encryption solution utilizing the CCSDS Space Data Link Security Protocol. This solution, named Key Management and Cryptography (KMC), incorporates JPL’s encryption service with the JSTAR open-source CryptoLib library, providing a solution for spacecraft command encryption, which is required per NASA standard NASA-STD-1006-A. This paper will focus on its design, usage, and operation for two SmallSat missions, along with its future expanded functionality of end-to-end spacecraft encryption.

command encryption↗
NASA NTRSDate not supplied

ARRISTOTLE Simulation Demo

NASA’s Katherine Johnson Independent Verification and Validation (NASA IV&V) Jon McBride Software Testing and Research (JSTAR) laboratory has developed the Advanced Risk Reduction Integrated Software Test and Operations Tri-program Lightweight Environment (ARRISTOTLE). ARRISTOTLE is a digital twin of NASA’s Artemis mission. ARRISTOLE executes the true SLS and Orion flight binaries integrated on a single laptop computer. This demo will showcase the execution of the full Artemis liftoff. Once airborne, an inflight anomaly will be injected, resulting in Orion detaching from the booster. The Launch Abort System (LAS) will safely bring Orion back to earth. The Unity game engine is used to visualize the flyout as flight software executes.

emulation↗
DOE OSTI2024

Latent Pitfalls in Microstructure-Based Modeling for Thermally Aged 9Cr-1Mo-V Steel (Grade 91)

A case study was conducted on a mechanistic model development that predicted tensile strength deterioration with thermal aging of 9Cr-1Mo-V steel in supporting the 60-year design life expected for advanced nuclear reactors. For property prediction beyond practical testing times, mechanistic modeling is highly desired, as it taps into the physics of structure–property relationships and therefore can generate reliable results for extrapolation. Meanwhile, as mechanistic models are often complicated, reflecting the intricacy of microstructure and strengthening mechanisms, pitfalls that are difficult to detect often exist. Here, this paper discusses latent pitfalls that are common in mechanistic modeling or specific in this 9Cr-1Mo-V case development through using the American Society of Mechanical Engineers verification and validation in computational solid mechanics (ASME V&V 10) standard for evaluating credibility of modeling in materials engineering. Suggestions are also made for enhancing reliability of microstructure-based modeling.

36 MATERIALS SCIENCE↗
NASA NTRS2016

Integrating MBSE into Ongoing Projects: Requirements Validation and Test Planning for the ISS SAFER

The International Space Station (ISS) Simplified Aid for Extra Vehicular Activity (EVA) Rescue (SAFER) is the spacewalking astronaut's final safety measure against separating from the ISS and being unable to return safely. Since the late 1990s, the SAFER has been a standard element of the spacewalking astronaut's equipment. The ISS SAFER project was chartered to develop a new block of SAFER units using a highly similar design to the legacy SAFER (known as the USA SAFER). An on-orbit test module was also included in the project to enable periodic maintenance/propulsion system checkout on the ISS SAFER. On the ISS SAFER project, model-based systems engineering (MBSE) was not the initial systems engineering (SE) approach, given the volume of heritage systems engineering and integration (SE&I) products. The initial emphasis was ensuring traceability to ISS program standards as well as to legacy USA SAFER requirements. The requirements management capabilities of the Cradle systems engineering tool were to be utilized to that end. During development, however, MBSE approaches were applied selectively to address specific challenges in requirements validation and test and verification (T&V) planning, which provided measurable efficiencies to the project. From an MBSE perspective, ISS SAFER development presented a challenge and an opportunity. Addressing the challenge first, the project was tasked to use the original USA SAFER operational and design requirements baseline, with a number of additional ISS program requirements to address evolving certification expectations for systems operating on the ISS. Additionally, a need to redesign the ISS SAFER avionics architecture resulted in a set of changes to the design requirements baseline. Finally, the project added an entirely new functionality for on-orbit maintenance. After initial requirements integration, the system requirements count was approaching 1000, which represented a growth of 4x over the original USA SAFER system. This presented the challenge - How to confirm that this new set of requirements set would result in the creation of the desired capability.

Anderson, Herbert A.↗