Search NASA⌕ Search

SEARCH · Search NASA

Results for “Run-Time Assurance”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

Towards Run-time Assurance of Advanced Propulsion Algorithms

This paper covers the motivation and rationale for investigating the application of run-time assurance methods as a potential means of providing safety assurance for advanced propulsion control systems. Certification is becoming increasingly infeasible for such systems using current verification practices. Run-time assurance systems hold the promise of certifying these advanced systems by continuously monitoring the state of the feedback system during operation and reverting to a simpler, certified system if anomalous behavior is detected. The discussion will also cover initial efforts underway to apply a run-time assurance framework to NASA's model-based engine control approach. Preliminary experimental results are presented and discussed.

run time assurance↗

Testing a Run-Time Assurance Framework Coupled with Integrated Risk Mitigation Capabilities for Autonomous Urban UAS Flights

The In-Time Aviation Safety Management System (IASMS) Concept of Operations (ConOps) envisions new capabilities to monitor, assess, and mitigate flight safety risks. Systems will be tailored to mission type, vehicle/equipage type, operational environment, and safety risk tolerance. Within an IASMS framework, several capabilities may be implemented spanning three operational phases (pre-flight, in-flight, and post-flight/off-line); and consisting of lower level functions and information services which may reside onboard the aircraft, on third-party server(s), and/or on ground/operator station(s). Each capability will be designed to produce and disseminate safety-relevant information; perform detection, diagnosis, and prediction of unsafe situations; and/or execute mitigation actions when hazardous events warrant such changes. This paper focuses on recent testing of airborne capabilities that demonstrate inflight aspects of the overarching concept for autonomous unmanned aircraft systems (UAS) operations in urban environments. A flight test architecture is described that applies run-time assurance principles (e.g., executes independent of the unassured autopilot), real-time risk assessment, and a technique to execute contingencies if necessary either automatically or via pilot intervention. Several tests using small UAS were conducted to verify the assured in-flight risk mitigation capability. The paper draws significantly from a larger NASA technical report and recent prior conference papers, providing additional details. Data is analyzed for two representative flights to illustrate the performance for various sequential and simultaneous hazards used during testing. During each automated flight, several hazards are encountered at various points along the flight path. At each point, the hazard is mitigated by the system, with the vehicle then continuing to subsequent points. The paper concludes with lessons-learned regarding relevant aspects of the overarching IASMS concept and how it may be updated and further advanced in the future.

population activity↗

Certification Strategies using Run-Time Safety Assurance for Part 23 Autopilot Systems

Part 23 aircraft operation, and in particular general aviation, is relatively unsafe when compared to other common forms of vehicle travel. Currently, there exists technologies that could increase safety statistics for these aircraft; however, the high burden and cost of performing the requisite safety critical certification processes for these systems limits their proliferation. For this reason, many entities, including the Federal Aviation Administration, NASA, and the US Air Force, are considering new options for certification for technologies that will improve aircraft safety. Of particular interest, are low cost autopilot systems for general aviation aircraft, as these systems have the potential to positively and significantly affect safety statistics. This paper proposes new systems and techniques, leveraging run-time verification, for the assurance of general aviation autopilot systems, which would be used to supplement the current certification process and provide a viable path for near-term low-cost implementation. In addition, discussions on preliminary experimentation and building the assurance case for a system, based on these principles, is provided.

automatic collision avoidance↗

Traveler Trustworthy Autonomy

NASAs Armstrong Flight Research Center has been engaged in the development of highly automatic safety systems for aviation since the mid 80s. For the past three years under Seedling and Center Innovation funding this work has moved toward the development of a software architecture applicable to autonomous safety. This work is now broadening and accelerating to address the airworthiness issues surrounding making a case for trustworthy autonomy. This software architecture is called the expandable variable-autonomy architecture (EVAA) and utilizes a run-time assurance approach to safety assurance.

autonomy↗

Leveraging ASTM Industry Standard F3269-17 for Providing Safe Operations of a Highly Autonomous Aircraft

This paper expands upon the ASTM industry standard F3269-17 to outline a run-time assurance (RTA) network architecture for use in ensuring safe flight operations of a highly autonomous aircraft. An RTA network architecture is proposed and critical features discussed to implement functions where automation is primarily responsible for the safety of the aircraft instead of a pilot. This shift in responsibility, made possible by the proposed architecture, is key to highly resilient automation and is a core enabler for future “pilotless” transportation concepts. The findings in this paper stem from the researcher’s experiences with ASTM in the generation of the standard and some seven years of RTA system development on various flight programs leveraging the RTA concepts outlined in the ASTM standard.

autonomous systems↗

Expandable Variable-Autonomy Architecture Viable Maneuver Displays

The Traveler-Resilient Autonomy team (RA) at NASA Armstrong Flight Research Center (AFRC) is developing a hierarchical autonomous system framework called EVAA– Expandable Variable-Autonomy Architecture which is a reference implementation of a Run-Time Assurance (RTA) architecture, that is seen as a likely path to the certification of autonomous Unmanned Aerial Systems (UAS).

Sarkis Serge Mikaelian↗

Design and Testing of an Approach to Automated In-Flight Safety Risk Management for sUAS Operations

An onboard risk management automation design is presented based on run-time assurance principles, as well as the concept for In-Time Aviation Safety Management Systems (IASMS) as described by the National Academies. The automation is designed to operate independently of the autopilot and perform real-time risk assessment spanning multiple classes of hazards, predict constraint violations, and track autopilot states. In the event of elevated risk conditions or predicted constraint violations, the automation will select from a set of available contingencies and trigger autopilot mode changes if necessary to mitigate risk exposure. The onboard automation also informs the remote operator/pilot of what the independent monitor is observing and any contingency decisions or actions that may arise during flight. Details of an implementation of this design and results of verification and validation activities, as required to meet stringent NASA software and system assurance standards, are also presented. This includes simulation and flight testing using small unmanned aircraft systems.

Ersin Ancel↗

Testing of Advanced Capabilities to Enable In-time Safety Management and Assurance for Future Flight Operations

In order to refine an initial Concept of Operations, explore Concepts of Use, and expose/validate requirements for future In-Time Aviation Safety Management Systems (IASMS), testing architectures were created, along with a set of capabilities and underlying information exchange protocols. These systems were conceived and developed based on hazards associated with two envisioned urban area flight domains: (1) highly autonomous small uncrewed aerial systems (sUAS) operating at low altitudes, and (2) highly autonomous air taxis. The initial scope of this development is described in [1]; this report provides an update, focusing on the subsequent developments and test activities. As stated in [1], it is important to note that there are many capabilities already in use by the industry (or soon to be in use) that will play critical roles in future IASMS designs. Those reported here were developed to address a gap in the current state-of-the-art regarding specific hazards/risks, and/or to allow for investigation of the interplay between and across hazard types — particularly regarding how overall safety risk can be reduced or managed effectively. Results of testing and development activities are organized by the operational phase wherein a particular capability would be employed (i.e., preflight, in-flight, and post-flight/off-line). Pre-flight: A set of capabilities were developed to help mitigate safety risk prior to flight (e.g., during flight and mission planning). Results of testing summarize (1) validation activities to raise the Technology Readiness Level (TRL) and (2) evaluation activities where the capabilities were applied to flight/mission planning procedures and used by operators/pilots. For the latter, flight plans were automatically assessed, and operators/pilots were notified of hazardous flight segments so as to enable adjustment of the flight plan and re-evaluation, and/or to better inform go/no-go decisions. Capabilities addressed hazards associated with power consumption, third-party risk, wind, navigation system performance, radiofrequency interference, and proximity to geo-spatial threats (e.g., buildings, trees, and no-fly zones). In-flight: Flight experiments tested capabilities that detect and respond to hazards encountered during flight. In the first series, safety hazards were monitored and assessed onboard, and system-generated mitigation maneuvers were recorded (but not acted upon by the vehicle). In the second series, mitigation maneuver commands directed the aircraft in response to safety hazards (i.e., auto-mitigation). The sUAS used for testing is described in full, as is the test architecture, which included commercial avionics, research avionics, and onboard software designed to detect, assess, and respond to hazards. The onboard system was designed as a run-time assurance framework, consistent with [2] and supportive of both supervisory and automated modes. The primary functions included: real-time risk assessment (RTRA), auto-pilot monitoring, constraint monitoring, and contingency select/triggering. RTRA performs integrated risk assessment considering data from several hazard-related monitors (e.g., battery, motors, navigation, communications, population density, and loss-of-control). Post-flight/off-line: Data monitored and recorded during flights can enable IASMS capabilities that execute after flights have completed (or “off-line”). These include: (1) the ability to identify anomalies and trends that may only be observable when comparing data spanning a number of similar flights; (2) the ability to update and validate pre-flight and in-flight capabilities and any underlying models to improve their performance; (3) the ability to report anomalies/off-nominals that may indicate design changes or maintenance actions are needed; and (4) the ability for humans involved in operations to report safety-relevant observations to help in understanding the flight data and/or the operational context of a flight. Progress on three such capabilities is summarized; the first investigates anomaly detection given a limited set of flight logs and applies an approach previously used for space operations. The second explores what could be identified using a larger set of flight logs, including from web-based forums where flight logs are posted by sUAS autopilot users. The third creates a new means of collecting information on UAS incidents and accidents via the Aviation Safety Reporting System (ASRS).

sUAS↗

Run Time Assurance for Electric Vertical Takeoff and Landing Aircraft

NASA is conducting research to demonstrate and evaluate the application of Run Time Assurance (RTA) as a means to assure safety in Electric Vertical Takeoff and Landing (eVTOL) aircraft with highly automated or autonomous flight capability supervised by a single onboard pilot. The work described in this report demonstrates an application of RTA and examines the implications for design and analysis of aircraft functions and systems; aircraft safety hazards; safety assurance; development assurance; and pilot tasks and performance. This research effort also seeks to assess the efficacy of the combined application of traditional Functional Hazard Analysis (FHA) and the more modern System Theoretic Process Analysis (STPA) techniques to perform hazard analyses on aircraft with complex automated and autonomous systems and an onboard pilot. During the research effort we developed architectural designs of two alternate eVTOL aircraft, generally following the process characterized in the SAE standards ARP4754 and ARP4761. The design has focused on the control architectures of these aircraft, which are identical except that one incorporates RTA techniques to reduce the criticality of some key software components. Artifacts of this process include a taxonomy of aircraft-level functions, aircraft-level architecture diagrams, aircraft-level functional hazard assessments (AFHA), function allocations onto aircraft systems and subsystems, functional block diagrams for a select set of control-related functions, and system-level functional hazard assessments (SFHA) for those functions. This project has highlighted the notion that DAL D is something of a sweet spot for low-confidence controllers in an RTA-based design. Among the many activities described in DO-178C, the activities related to requirement verifiability, algorithmic accuracy, and test coverage can be the most challenging for the kinds of advanced control techniques that may be desirable in novel UAM designs, such as adaptive control, machine-learning, artificial intelligence, numerical search, and Monte Carlo based algorithms. Moreover, the standard requires that development teams demonstrate that errors leading to unacceptable failure conditions have been removed from the software. The RTA architecture, which cordons off the low-confidence function, makes it much easier to show this for these kinds of algorithms. With regard to the use of STPA and FHA as complementary hazard analysis techniques, our research effort led us to the conclusion that STPA should be used to derive requirements for hardware and software systems and/or components. Also, STPA is a natural complement to other processes in ARP4754A involving design studies and iteration.

Run-time assurance↗

Flight Testing of In-Time Safety Assurance Technologies for UAS Operations

Ongoing research at NASA is driven by a strategic plan defined by the Aeronautics Research Mission Directorate and a vision for future In-Time Aviation Safety Management Systems (IASMS) as described by the National Academies. In both visions, system safety awareness and provision are expanded through increased access to relevant data; integrated analysis and predictive capabilities; improved real-time detection and alerting of domain-specific hazards; decision support, and in some cases, automated risk mitigation strategies. One primary research focus is to develop means by which more timely (i.e., “in-time”) actions may be taken to mitigate precursors, anomalies, or trends that are observed during operations. In this paper, we describe such means as a collection of Services, Functions, and Capabilities (SFCs) that are supported by an underlying information system. For example, an integrated risk assessment capability is envisioned that continuously monitors safety-related metrics and margins and recommends timely operational changes. Assessment functions and/or services can be based on data analytics and predictive models derived from heterogeneous data sets that span relevant indicator metrics and their time histories. Likewise, on-board functions can identify and reduce susceptibility to precursor conditions that have led (and can lead) to aircraft loss-of-control or out-of-control accidents. This paper summarizes development and testing of such an information system tailored to hazards anticipated for future highly autonomous flight missions near and over densely populated areas. Testing is accomplished via simulation and by using small, unmanned aircraft operating over a test range at NASA’s Langley Research Center. Flight plans and test scenarios are defined to emulate several use-cases, including package delivery; reconnaissance; fire management; and urban air taxi vertiport operations. Two test phases are summarized with Phase 1 occurring in (2019-2020) and Phase 2 ongoing (2021-present). Results focus on SFC performance, technology readiness level assessment, and requirements discovery/validation. Companion papers are cited throughout for additional details on the recent testing.

safety management↗

Advanced Software V&V for Civil Aviation and Autonomy

With the advances in high-computing platform (e.g., advanced graphical processing units or multi-core processors), computationally-intensive software techniques such as the ones used in artificial intelligence or formal methods have provided us with an opportunity to further increase safety in the aviation industry. Some of these techniques have facilitated building safety at design time, like in aircraft engines or software verification and validation, and others can introduce safety benefits during operations as long as we adapt our processes. In this talk, I will present how NASA is taking advantage of these new software techniques to build in safety at design time through advanced software verification and validation, which can be applied earlier and earlier in the design life cycle and thus help also reduce the cost of aviation assurance. I will then show how run-time techniques (such as runtime assurance or data analytics) offer us a chance to catch even more complex problems, even in the face of changing and unpredictable environments. These new techniques will be extremely useful as our aviation systems become more complex and more autonomous.

software assurance↗

Unique NISTAR-Based Climate GCM Diagnostics of the Earth’s Planetary Albedo and Spectral Absorption Through Longitudinal Data Slicing

Deep Space Climate Observatory (DSCOVR) measurements of Earth’s reflected solar and emitted thermal radiation permit a unique model/data comparison perspective that is not readily available from other satellite data. The key factor is the unique Lissajous orbital viewing geometry from the Lagrangian L1 point, which enables a continuous view of Earth’s sunlit hemisphere. The National Institute of Standards and Technology Advanced Radiometer (NISTAR) is the DSCOVR Mission energy budget instrument, which views the reflected and emitted radiation of the Earth’s sunlit hemisphere by means of single pixel active cavity full-spectrum (Band-A, 0.2–100 μm) and filtered solar wavelength (Band-B, 0.2–4.0 μm; and Band-C, 0.7–4.0 μm) radiometer measurements. An additional solar wavelength photodiode channel (0.3–1.1 μm) provides a calibration reference. The objective of this study is the assessment of climate GCM performance via direct model/data comparisons. Such comparisons are difficult due to quasi-chaotic natural variability present in real-world observational data and in climate GCM simulations. This is where the unique DSCOVR viewing geometry makes possible the longitudinal data slicing methodology for more direct model/data comparison. The key point of the longitudinal slicing approach is that data integration over the entire sunlit hemisphere eliminates the quasi-chaotic meteorological weather-scale noise, while preserving intra-seasonal and planetary-scale variability. The rotation of the Earth that retrieves this climate-style, large-scale longitudinal and seasonal variability. The hemispheric averaging is accomplished automatically in NISTAR measurements with its single-pixel view of the Earth. For climate GCMs, this requires implementing the Sunlit Hemisphere Sampling (SHS) scheme to operate on the GCM run-time output data, utilizing the DSCOVR Satellite Ephemeris data to assure precise viewing geometry between NISTAR measurements and GCM output data, while averaging out the meteorological weather noise. However, GCM generated data are radiative fluxes, while NISTAR (and EPIC) measurements are near-backscattered radiances. Conversing NISTSR measurements into radiative fluxes cannot be accomplished using NISTAR data alone, even with detailed support from conventional satellite data. But the identical viewing geometry of Earth’s sunlit hemisphere, and synergistic analyses of EPIC data make it feasible for this conversion of NISTAR near-backscatter radiances into radiative fluxes.

deep space climate observatory↗

Reliability-Aware Requirements Development for Autonomy Software

Autonomy is the ability of a system to make decisions and take actions in the presence of uncertainty. Validating autonomy, therefore, is a matter of validating that the system makes intelligent decisions even when there might be discrepancies between the observed and the expected state of the world as well as when the possible outcome of each of the possible courses of action is uncertain. During the requirements engineering phase, the functions, including those that are required to be autonomous and their expected behavior are specified. However, in the case of autonomous software system, it is challenging to specify all possible scenarios that the software may encounter. Additionally, since autonomous behaviors cannot always be fully predetermined, it is difficult to reason about their completeness and correctness. While these challenges are also faced with non-autonomous system, they are more prevalent for autonomous system due to their complexity and emergent behavior. Our efforts have been three-fold. On one hand, we have developed a goal-based method for requirements decomposition. We have embedded this approach into a higher-level systems engineering framework and then developed methods for decision and reliability analysis within this framework. And finally, we use simulation analysis to validate the more theoretical methods and provide a feedback mechanism. The decision analysis approach (DA) helps in identifying and mitigating the run-time risks, by bringing to the forefront the uncertainties, decisions, interactions, and other factors that may cause autonomous software to make erroneous decisions. The DA framework is used to quantify, for each decision that the software can make, failure risk due to uncertainties. The information about the key decisions and circumstances that can cause autonomous system to make incorrect decisions are used to generate test scenarios to be run in a simulator to ensure that the system can handle error-prone circumstances. The results or outcomes from decisions, based on the simulated scenarios, can then be leveraged to further gain insight of the assured system. We combine a goal-based approach with simulation analysis to facilitate requirements development for autonomous systems and further provide a method for run-time systems level reliability considerations during the autonomous software development process. The DA and the reliability analysis methods provide a formalism for the consideration of uncertainty in the run time decision-making process for autonomous software. This formalism considers the uncertainty involved in the outcome of different courses of action, in terms of performance and cost, as well as a method to consider the system level reliability implications. System level reliability considerations for autonomous software decision making contribute to the satisfaction of the system level goals over the lifecycle of a given mission. We demonstrate our approach using a hypothetical rover path planning example. The requirements for our rover simulator are based on existing literature about Mars rovers and lessons learned from real world observations at JPL. The specifics of the autonomy design for the Mars rovers, that have not been previously cleared for external release, are not used for our demonstrations.

Lindvall, Mikael↗

Survey of Verification and Validation Techniques for Small Satellite Software Development

The purpose of this paper is to provide an overview of the current trends and practices in small-satellite software verification and validation. This document is not intended to promote a specific software assurance method. Rather, it seeks to present an unbiased survey of software assurance methods used to verify and validate small satellite software and to make mention of the benefits and value of each approach. These methods include simulation and testing, verification and validation with model-based design, formal methods, and fault-tolerant software design with run-time monitoring. Although the literature reveals that simulation and testing has by far the longest legacy, model-based design methods are proving to be useful for software verification and validation. Some work in formal methods, though not widely used for any satellites, may offer new ways to improve small satellite software verification and validation. These methods need to be further advanced to deal with the state explosion problem and to make them more usable by small-satellite software engineers to be regularly applied to software verification. Last, it is explained how run-time monitoring, combined with fault-tolerant software design methods, provides an important means to detect and correct software errors that escape the verification process or those errors that are produced after launch through the effects of ionizing radiation.

Small Satellite Technology↗

Improving system reliability through formal analysis and use of checks in software

Software is playing increasingly important roles in avionics systems. It is widely used in navigation and, in some cases, in control loops that maintain aircraft stability. To guarantee the safety of flight systems, the FAA requires that critical components have a probability of failure no greater than 10(exp -9) per hour of flight. Software is being used to diagnose system components for failure. SIFT (Software Implemented Fault Tolerance) was a computer system developed to study the use of software to check for failure and manage processor reconfiguration. To guarantee that software satisfies its specifications, formal verification can be used. With this a program and its specification are viewed as mathematical objects, and a mathematical proof is used to show that the program and its specification are equivalent. In previous research, a theory of checking was developed to offer assistance in analyzing specifications and designing run-time checks. In the theory, checking is considered abstractly in terms of n-ary relations much like those of relational database theory. Within the theory check are categorized, checks on input and checks on results are considered, and formal attention is given to the minimization and logical combination of checks. The focus is upon input checks and the obstacles in checking input to critical systems. A central concern is with a property referred to as independence. The concern is with circumstances under which it is possible to apply isolated, independent checks to separate sensor inputs and be assure that all illegal input will be properly detected. Presently, independence is being investigated and checked in the context of the GCS (Guidance and Control System). The GCS simulator is intended for testing software that implements control laws for landing spacecraft. The large number of inputs and their complex interrelationships provide an exciting context in which to investigate independence and the difficulties of supplying input checks.

Staknis, Mark E.↗