Whisper Program Suite Validation and Verification Report
Abstract not provided.
SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Abstract not provided.
Testing has long been recognized as a critical component of spacecraft development activities - yet many major systems failures may have been prevented with more rigorous testing programs. The question is why is more testing not being conducted? Given unlimited resources, more testing would likely be included in a spacecraft development program. Striking the right balance between too much testing and not enough has been a long-term challenge for many industries. The objective of this paper is to discuss some of the barriers, enablers, and best practices for developing and sustaining a strong test program and testing team. This paper will also explore the testing decision factors used by managers; the varying attitudes toward testing; methods to develop strong test engineers; and the influence of behavior, culture and processes on testing programs. KEY WORDS: Risk, Integration and Test, Validation, Verification, Test Program Development
An algorithm for statistical verification of the XT26 code contained in the SAPHIRE-2006 code system is described. The results of conservative estimation of the calculation error in the K{sub eff} calculations for different types of benchmark experiments are presented. The results of the statistical analysis of deviations from the experimental values are compared with the corresponding parameters obtained from the set of calculations performed using other codes.
The authors present an initial and elementary investigation of the formal specification and mechanical verification of programs that interact with environments. They describe a mechanical proof that a simple, real time control program keeps a vehicle on a straightline course in a variable crosswind. To formalize the specification they define a mathematical function which models the interaction of the program and its environment. They then state and proved two theorems about this function: the simulated vehicle never gets farther than three units away from the intended course, and it comes to the course if the wind ever remains steady for at least four sampling units.
Trustworthy savings calculations are critical to convincing regulators of both the cost-effectiveness of energy efficiency program investments and their ability to defer supply-side capital investments. Today’s methods for measurement and verification (M&V) of energy savings constitute a significant portion of the total costs of energy efficiency programs. They also require time-consuming data acquisition. A spectrum of savings calculation approaches is used, with some relying more heavily on measured data and others relying more heavily on estimated, modeled, or stipulated data. The increasing availability of “smart” meters and devices that report near-real time data, combined with new analytical approaches to quantify savings, offers the potential to conduct M&V more quickly and at lower cost, with comparable or improved accuracy. Commercial energy management and information systems (EMIS) technologies are beginning to offer these ‘M&V 2.0’ capabilities, and program administrators want to understand how they might assist programs in quickly and accurately measuring energy savings. This paper presents the results of recent testing of the ability to use automation to streamline the M&V process. In this paper, we apply an automated whole-building M&V tool to historic data sets from energy efficiency programs to begin to explore the accuracy, cost, and time trade-offs between more traditional M&V, and these emerging streamlined methods that use high-resolution energy data and automated computational intelligence. For the data sets studied we evaluate the fraction of buildings that are well suited to automated baseline characterization, the uncertainty in gross savings that is due to M&V 2.0 tools’ model error, and indications of labor time savings, and how the automated savings results compare to prior, traditionally determined savings results. The results show that 70% of the buildings were well suited to the automated approach. In a majority of the cases (80%) savings and uncertainties for each individual building were quantified to levels above the criteria in ASHRAE Guideline 14. In addition the findings suggest that M&V 2.0 methods may also offer time-savings relative to traditional approaches. Lastly, we discuss the implications of these findings relative to the potential evolution of M&V, and pilots currently being launched to test how M&V automation can be integrated into ratepayer-funded programs and professional implementation and evaluation practice.
BACKGROUND At NASA, verification testing is the formal process of ensuring that a product conforms to requirements set by a project or program. Some verification methods, such as Demonstrations and Test, require either the end product or a mockup of the product with sufficient fidelity to stand-in for the product during the test. Traditionally, these mockups have been physical (e.g., foam-core and wood) but there is growing interest in exploring new methods for testing with these mockups. These methods include virtual reality (VR), mixed reality, and augmented reality which are collectively referred to as eXtended Reality (XR) technologies. VR has already been adopted and used by many in the aerospace industry as a tool for use in early design phases (e.g., developmental testing) and may have the most potential for use in verification tests. Benefits of using VR mockups offer cost effectiveness, ease of iteration, simulation of hazardous conditions (e.g., an egress through a hatch with smoke obscuring vision), and the ability to simulate microgravity conditions, which are challenging to do with physical mockups. However, the validity of test results obtained from VR mockup demonstrations or testing, compared to the current gold standard of physical mockups, remains uncertain. It is unlikely that there is one clean answer as there are many different types of verification outcomes and each XR technology must be evaluated on its own merits. This is not an issue during developmental testing as the design is still in flux and the total success of the design is not dependent upon the results of a developmental test. Verification tests, however, only happen once, assuming no change to the design, and the results are used to certify the product. Therefore, establishing the validity of XR mockup-based verification outcomes is essential before considering them for any use in verification tests. OBJECTIVE AND METHOD To address this concern, the Human Research Program has funded a project to explore and qualify how XR technologies might be used in verification demonstration and testing at NASA. Currently, we are conducting a review of the literature on the utilization of XR mockups for design activities, prototyping, and user testing. We are employing the Strengths, Weaknesses, Opportunities, and Threats (SWOT) analysis method to identify the pros, cons, and barriers to adoption of XR technologies for verification testing at NASA. Additionally, we are developing a framework to guide the deployment of XR mockups for verification tests. Building upon available evidence from the literature and subject-matter expert feedback, our goal for the framework is to provide guidelines for which forms of XR mockups are suitable for a given verification test, when only physical mockups should be employed and to highlight areas for which more evidence is needed. To further refine our framework and to contribute to the body of evidence, we are planning a lab-based experiment comparing a VR mockup to a physical twin for a set of select verification outcomes. ANTICIPATED RESULTS In this presentation, we will present the work we conducted to evaluate XR technologies for use in verification tests at NASA. We will summarize and report our findings from the SWOT analysis and our lab-based study, and we will present the current state of the XR Technologies for Verification Testing framework. We will conclude by summarizing remaining work and future directions for the project. Technologies for Verification Testing framework. We will conclude by summarizing remaining work and future directions for the project.
Contents include the following: 1. Hyper-X program: Propulsion development and verification. 2. GTX program: Airbreathing launch vehicles. 3. Hypersonic technology development: Technology program overviews. Ramjet/scramjet research. 4. Hypersonic test methods: Test medium effects. 5. Advanced propulsion: RBCC engine design and performance assessments. Advanced and combined cycle engine technology.
We present a framework, called AIR, for verifying safety properties of assembly language programs via software model checking. AIR extends the applicability of predicate abstraction and counterexample guided abstraction refinement to the automated verification of low-level software. By working at the assembly level, AIR allows verification of programs for which source code is unavailable-such as legacy and COTS software-and programs that use features-such as pointers, structures, and object-orientation-that are problematic for source-level software verification tools. In addition, AIR makes no assumptions about the underlying compiler technology. We have implemented a prototype of AIR and present encouraging results on several non-trivial examples.
Functional fault models (FFMs) are a directed graph representation of the failure effect propagation paths within a system's physical architecture and are used to support development and real-time diagnostics of complex systems. Verification of these models is required to confirm that the FFMs are correctly built and accurately represent the underlying physical system. However, a manual, comprehensive verification process applied to the FFMs was found to be error prone due to the intensive and customized process necessary to verify each individual component model and to require a burdensome level of resources. To address this problem, automated verification tools have been developed and utilized to mitigate these key pitfalls. This paper discusses the verification of the FFMs and presents the tools that were developed to make the verification process more efficient and effective.
The C programming language is widely used for programming space flight software and other safety-critical real time systems. C, however, is far from ideal for this purpose: as is well known, it is both low-level and unsafe. This paper describes Spot, a language derived from C for programming space flight systems. Spot aims to maintain compatibility with existing C code while improving the language and supporting verification with the SPIN model checker. The major features of Spot include actor-based concurrency, distributed state with message passing and transactional updates, and annotations for testing and verification. Spot also supports domain-specific annotations for managing spacecraft state, e.g., communicating telemetry information to the ground. We describe the motivation and design rationale for Spot, give an overview of the design, provide examples of Spot's capabilities, and discuss the current status of the implementation.
This Final Technical Report summarizes the research work conducted under NASA's Physical Oceanography Program entitled, Verification And Improvement Of ERS-112 Altimeter Geophysical Data Recorders For Global Change Studies, for the time period from January 1, 2000 through June 30, 2000. This report also provides a summary of the investigation from July 1, 1997 - June 30, 2000. The primary objectives of this investigation include verification and improvement of the ERS-1 and ERS-2 radar altimeter geophysical data records for distribution of the data to the ESA-approved U.S. ERS-1/-2 investigators for global climate change studies. Specifically, the investigation is to verify and improve the ERS geophysical data record products by calibrating the instrument and assessing accuracy for the ERS-1/-2 orbital, geophysical, media, and instrument corrections. The purpose is to ensure that the consistency of constants, standards and algorithms with TOPEX/POSEIDON radar altimeter for global climate change studies such as the monitoring and interpretation of long-term sea level change. This investigation has provided the current best precise orbits, with the radial orbit accuracy for ERS-1 (Phases C-G) and ERS-2 estimated at the 3-5 cm rms level, an 30-fold improvement compared to the 1993 accuracy. We have finalized the production and verification of the value-added ERS-1 mission (Phases A, B, C, D, E, F, and G), in collaboration with JPL PODAAC and the University of Texas. Orbit and data verification and improvement of algorithms led to the best data product available to-date. ERS-2 altimeter data have been improved and we have been active on Envisat (2001 launch) GDR algorithm review and improvement. The data improvement of ERS-1 and ERS-2 led to improvement in the global mean sea surface, marine gravity anomaly and bathymetry models, and a study of Antarctica mass balance, which was published in Science in 1998.
This brief seeks to inform Commercial Property Assessed Clean Energy (C-PACE) program administrators about design and execution of measurement and verification (M&V) standards by leveraging the experience of existing C-PACE programs. It also serves as a resource for state and local jurisdictions interested in establishing new C-PACE programs that incorporate M&V standards. It defines M&V standards in broad terms as technical standards to verify and demonstrate performance of C-PACE projects and programs, regardless of whether the programs require performance guarantees or ongoing post-project M&V.
The purpose of this document is to define the process for validation and verification (V&V) of software used to test the thermal data acquisition equipment in the Oak Ridge National Laboratory (ORNL) Package Testing Program (PTP) as part of the testing configuration(s). This process adheres to all applicable ORNL Standards Based Management System requirements and correlates with the PTP Quality Management System and the Software Quality Assurance Plan.
After implementing several energy efficiency standards and labels (30 products covered by MEPS, 50 products covered by voluntary labels and 19 products by mandatory labels), the China National Institute of Standardization (CNIS) is now implementing verification and compliance mechanism to ensure that the energy information of labeled products comply with the requirements of their labels. CNIS is doing so by organizing check testing on a random basis for room air-conditioners, refrigerators, motors, heaters, computer displays, ovens, and self -ballasted lamps. The purpose of the check testing is to understand the implementation of the Chinese labeling scheme and help local authorities establishing effective compliance mechanisms. In addition, to ensure robustness and consistency of testing results, CNIS has coordinated a round robin testing for room air conditioners. Eight laboratories (Chinese (6), Australian (1) and Japanese (1)) have been involved in the round robin testing and tests were performed on four sets of samples selected from manufacturer?s production line. This paper describes the methodology used in undertaking both check and round robin testing, provides analysis of testing results and reports on the findings. The analysis of both check and round robin testing demonstrated the benefits of a regularized verification and monitoring system for both laboratories and products such as (i) identifying the possible deviations between laboratories to correct them, (ii) improving the quality of testing facilities, (iii) ensuring the accuracy and reliability of energy label information in order to strength the social credibility of the labeling program and the enforcement mechanism in place.
After implementing several energy efficiency standards and labels (30 products covered by MEPS, 50 products covered by voluntary labels and 19 products by mandatory labels), the China National Institute of Standardization (CNIS) is now implementing verification and compliance mechanism to ensure that the energy information of labeled products comply with the requirements of their labels. CNIS is doing so by organizing check testing on a random basis for room air-conditioners, refrigerators, motors, heaters, computer displays, ovens, and self -ballasted lamps. The purpose of the check testing is to understand the implementation of the Chinese labeling scheme and help local authorities establishing effective compliance mechanisms. In addition, to ensure robustness and consistency of testing results, CNIS has coordinated a round robin testing for room air conditioners. Eight laboratories (Chinese (6), Australian (1) and Japanese (1)) have been involved in the round robin testing and tests were performed on four sets of samples selected from manufacturer's production line. This paper describes the methodology used in undertaking both check and round robin testing, provides analysis of testing results and reports on the findings. The analysis of both check and round robin testing demonstrated the benefits of a regularized verification and monitoring system for both laboratories and products such as (i) identifying the possible deviations between laboratories to correct them, (ii) improving the quality of testing facilities, (iii) ensuring the accuracy and reliability of energy label information in order to strength the social credibility of the labeling program and the enforcement mechanism in place.
At Pennsylvania Department of Environmental Protection's request, ORAU's IEAV program conducted verification surveys on the excavated surfaces of Section 3, SUs 1, 4, and 5 at the Whittaker site on March 13 and 14, 2013. The survey activities included visual inspections, gamma radiation surface scans, gamma activity measurements, and soil sampling activities. Verification activities also included the review and assessment of the licensee's project documentation and methodologies. Surface scans identified four areas of elevated direct gamma radiation distinguishable from background; one area within SUs 1 and 4 and two areas within SU5. One area within SU5 was remediated by removing a golf ball size piece of slag while ORAU staff was onsite. With the exception of the golf ball size piece of slag within SU5, a review of the ESL Section 3 EXS data packages for SUs 1, 4, and 5 indicated that these locations of elevated gamma radiation were also identified by the ESL gamma scans and that ESL personnel performed additional investigations and soil sampling within these areas. The investigative results indicated that the areas met the release criteria.
The Navigation System on the NASA Space Launch System (SLS) Block 1 vehicle performs initial alignment of the Inertial Navigation System (INS) navigation frame through gyrocompass alignment (GCA). Because the navigation architecture for the SLS Block 1 vehicle is a purely inertial system, the accuracy of the achieved orbit relative to mission requirements is very sensitive to initial alignment accuracy. The assessment of this sensitivity and many others via simulation is a part of the SLS Model-Based Design and Model-Based Requirements approach. As a part of the aforementioned, 6DOF Monte Carlo simulation is used in large part to develop and demonstrate verification of program requirements. To facilitate this and the GN&C flight software design process, an SLS-Program-controlled Design Math Model (DMM) of the SLS INS was developed by the SLS Navigation Team. The SLS INS model implements all of the key functions of the hardware-namely, GCA, inertial navigation, and FDIR (Fault Detection, Isolation, and Recovery)-in support of SLS GN&C design requirements verification. Despite the strong sensitivity to initial alignment, GCA accuracy requirements were not verified by test due to program cost and schedule constraints. Instead, the system relies upon assessments performed using the SLS INS model. In order to verify SLS program requirements by analysis, the SLS INS model is verified and validated against flight hardware. In lieu of direct testing of GCA accuracy in support of requirement verification, the SLS Navigation Team proposed and conducted an engineering test to, among other things, validate the GCA performance and overall behavior of the SLS INS model through comparison with test data. This paper will detail dynamic hardware testing of the SLS INS, conducted by the SLS Navigation Team at Marshall Space Flight Center's 6DOF Table Facility, in support of GCA performance characterization and INS model validation. A 6-DOF motion platform was used to produce 6DOF pad twist and sway dynamics while a simulated SLS flight computer communicated with the INS. Tests conducted include an evaluation of GCA algorithm robustness to increasingly dynamic pad environments, an examination of GCA algorithm stability and accuracy over long durations, and a long-duration static test to gather enough data for Allan Variance analysis. Test setup, execution, and data analysis will be discussed, including analysis performed in support of SLS INS model validation.