Trusted Autonomy: Autonomy and Autonomous Systems at NASA LaRC's Autonomy Incubator
No abstract available
SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
No abstract available
The Distributed Spacecraft Autonomy project is developing a suite of software tools that enable an operator to command and receive data from a swarm as a single entity, enable a swarm to autonomously coordinate its actions via distributed decision making and reactive closed-loop control, and model swarm behavior in the presence of anomalies or failures. Our use case is the mapping of the electron density of the ionosphere using radio tomography by coordinating the selection of appropriate GPS channels, and by recording Total Electron Count (TEC) measurements. DSA will be demonstrated onboard the NASA Ames Starling mission – a swarm of four small, LEO spacecraft, scheduled to launch in 2021. We will also perform a ground demonstration with simulated and hardware-in-the-loop elements, to validate the tools for controlling swarms of up to 100 assets. The capability to communicate autonomously between the swarm satellites is demonstrated via a sophisticated simulation architecture. Historical Plasmasphere TEC data obtained via dual-band Novatel GPS Receivers are utilized as a representative input dataset for the swarm. The representative TEC data and GPS satellite observability information is fed to the autonomous software package in place of a true real-time ground data collection process. The swarm satellites actively share status updates amongst one another and utilize multi-agent decision making to optimally identify regions of interest in the TEC distribution. The software, aware of the bandwidth limitations of the swarm satellites, prioritizes explorative measurements, which define the range of observability for the satellites, as well as exploitative measurements, which focus on maximizing the observance potential of regions with prolonged, elevated TEC density. The science of this study can ultimately be used to determine the dynamics and coupling of Earth’s magnetosphere, ionosphere, and atmosphere and their response to solar and terrestrial inputs. The findings can be applied to the imaging of critical, transient phenomena in the magnetosphere in later missions. Meanwhile, the swarm autonomy capabilities have far reaching potential in future satellite missions. As an experimental demonstration of the autonomous capabilities of the network, a message is first printed within a core Flight Executive (cFE) application. Two cFE applications that communicate with one another within the same core Flight System (cFS) are shown. Communication between mission applications on the internal cFE bus is extended to utilize Data Distribution Service (DDS) for vehicle-to-vehicle networking. The DDS middleware provides reliable delivery, routing, and topic subscription features over User Datagram Protocol (UDP). Leveraging Linux containerization, a networked set of satellite instances are generated by script to simulate swarm behavior. Swarm commanding and synchronization through the network is demonstrated under various topologies and data-loss conditions. Finally, autonomous swarm scalability from 2 satellites to 100 satellites is shown.
The Distributed Spacecraft Autonomy project is developing a suite of software tools that enable an operator to command and receive data from a swarm as a single entity, enable a swarm to autonomously coordinate its actions via distributed decision making and reactive closed-loop control, and model swarm behavior in the presence of anomalies or failures. Our use case is the mapping of the electron density of the ionosphere using radio tomography by coordinating the selection of appropriate GPS channels, and by recording Total Electron Count (TEC)measurements. DSA will be demonstrated on board the NASA Ames Starling mission a swarm of four small, LEO spacecraft, scheduled to launch in 2021. We will also perform a ground demonstration with simulated and hardware-in-the-loop elements, to validate the tools for controlling swarms of up to 100 assets.The capability to communicate autonomously between the swarm satellites is demonstrated via a sophisticated simulation architecture. Historical Plasma sphere TEC data obtained via dual-band Novatel GPS Receivers are utilized as a representative input data set for the swarm. The representative TEC data and GPS satellite observability information is fed to the autonomous software package in place of a true real-time ground data collection process. The swarm satellites actively share status updates amongst one another and utilize multi-agent decision making to optimally identify regions of interest in the TEC distribution. The software,aware of the bandwidth limitations of the swarm satellites, prioritizes explorative measurements,which define the range of observability for the satellites, as well as exploitative measurements,which focus on maximizing the observance potential of regions with prolonged, elevated TEC density. The science of this study can ultimately be used to determine the dynamics and coupling of Earth's magnetosphere, ionosphere, and atmosphere and their response to solar and terrestrial inputs. The findings can be applied to the imaging of critical, transient phenomena in the magnetosphere in later missions. Meanwhile, the swarm autonomy capabilities have far reaching potential in future satellite missions.As an experimental demonstration of the autonomous capabilities of the network, a message is first printed within a core Flight Executive (cFE) application. Two cFE applications that communicate with one another within the same core Flight System (cFS) are shown.Communication between mission applications on the internal cFE bus is extended to utilize Data Distribution Service (DDS) for vehicle-to-vehicle networking. The DDS middle ware provides reliable delivery, routing, and topic subscription features over User Data gram Protocol (UDP).Leveraging Linux containerization, a networked set of satellite instances are generated by script to simulate swarm behavior. Swarm commanding and synchronization through the network is demonstrated under various topologies and data-loss conditions. Finally, autonomous swarms calability from 2 satellites to 100 satellites is shown.
Explore the source record for details and available documents.
Explore the source record for details and available documents.
2024 ASCEND Call for Sessions Session Format: Panel Session Topic: Space Exploration and Infrastructure: Exploring, Living, and Working in Space (The panel must map to one of six Session Topics - https://www.ascend.events/presenters/call-for-sessions/#sessiontopics) Title: Autonomy to Enable NASA Missions from Aeronautics to Space Short Session Description: In this panel discussion the National Aeronautics and Space Administration (NASA) will discuss the role that autonomy and Artificial Intelligence (AI) will play as humanity moves off-world. Recent advances in general autonomy tools are changing the way NASA and its partners leverage autonomy for its air and space initiatives, including Advanced Air Mobility (AAM) concepts and potential lunar and Martian operations. The panelists will consist of autonomy subject matter experts familiar with the current state of the art for autonomy across both aeronautical and space domains. They will discuss how those technologies could evolve as operations become more complex and which autonomy technologies can be used in both the space and aeronautical domains. For example, perhaps autonomy work originally developed for terrestrial applications, like Advanced Air Mobility (AAM), could be applied to off-world lunar and Martian applications and vice versa. Additionally, the panel will address common misconceptions of these technologies, obstacles to implementation, and possible solutions for overcoming those obstacles. Join NASA in exploring how research activities can align to streamline autonomy development efforts, advancing NASA's goals to expand humanity's reach beyond Earth for the benefit of all. Contact Information: • Dr. Adam Yingling • Office of Technology, Policy, and Strategy (OTPS) • Adam.j.yingling@nasa.gov, 703-416-9129 Session Length: 1.25 hours, 75 minutes Extended Description: Moderator: Dr. Adam Yingling, NASA, Office of Technology, Policy, and Strategy Panel Speakers: • Autonomy Forum Principals o Dr. Charles Norton, Deputy Chief Technologist, Jet Propulsion Laboratory (JPL) o Dr. Carolyn Mercer, Chief Technologist, Space Mission Directorate (SMD) o Dr. Parimal Kopardekar (PK), Advanced Air Mobility (AAM) Integration Manager o Danette Alan, NASA, Senior Leader for Autonomous Systems, Space Technology Mission Directorate (STMD) o Duane Armstrong, Intelligent Systems Lead, Autonomous Systems Laboratory (ASL) Panel Format: • Introduction (10 min): The panel moderator will provide a 10-minute session introduction that will include an overview NASA’s Moon to Mars architecture, AAM autonomy research, and an introduction of the principals as panel speakers. • Moderated Session Part 1 (30 min): There will be a 30-minute moderated session among the moderator and the five autonomy principals to discuss the current state of art for autonomy and how work developed in one domain may be applicable to other domains; including the merits and challenges for implementing those technologies. • Moderated Session Part 2 (25 min): The moderator will then ask the principals to consider how technologies developed across all the domains might be able to address the most salient challenges identified in the previous session. • Q&A (10 min): The session will conclude with 10-minutes of audience Q&A. Session Goals and Outcomes: The session goals are 1) to communicate the importance of autonomy for both aeronautical and space mission, 2) Investigate potential synergies across autonomy research efforts that will enable scalable operations, and 3) to receive community feedback as NASA leverages autonomy to evolve aviation on Earth and enable humanity to live and work off-world.
To achieve NASA’s Artemis program mission objectives a high level of autonomy and ubiquitous autonomy throughout the systems that are being developed will be necessary. The autonomous systems of Artemis will require a distributed autonomy capability, with autonomous systems organized functionally in a hierarchical architecture, where systems at higher levels of the hierarchy have authority over systems at lower levels. The challenge of developing autonomy technologies and concepts of operations for Artemis has been undertaken by the NASA Gateway Working Group. This group has developed requirements, architectures, concepts of operations, and interface control documents, in the context of a hierarchical distributed architecture that includes the following: a Vehicle System Manager (VSM) that autonomously manages the entire Gateway; Module System Managers (MSMs) that autonomously manage each module; and System Managers (SMs) that autonomously manage systems within a module(i.e. ECLSS).A substantially high level of autonomy needs to be achieved by each element of the hierarchy (VSM, MSM, SM)to meet requirements for uncrewed operations; this includes conditions that will have minimal and/or delayed ground intervention (i.e. requirements for sustainability for months of operation without crew or ground support). To advance an implementation of this autonomy design (Gateway Autonomy Design –GAD), a collaboration was established between the Autonomous Systems Laboratory (ASL) at NASA Stennis Space Center and Lockheed Martin. The objectives of this partnership were the following: (1)to implement autonomy at the VSM, MSM, and SM levels;(2) to implement communications among a VSM, 2 MSMs, ORION (a visiting vehicle somewhat equivalent to a module) and 1 SM (a power system), and (3) test autonomous operations with representative use cases. A SM backed by a high-fidelity simulation was created to facilitate demonstrations of use cases that originated in a system of a module. Communication between the VSM and MSMs was implemented according to Concepts of Operations and Interface Control Documents (ICDs). Demonstrations were conducted to address nominal and off-nominal operations and multi-module interactions with VSM. Additionally, user interfaces were created to provide awareness about ongoing processes and results while enhancing the demonstration. Demonstrations included the following use cases: (1)Orion as visiting vehicle registers with VSM;(2) VSM reschedules a module’s timelines when another module’s MSM task fails; and (3) a module’s Power System Manager (PSM) standalone demonstration that included component failure diagnostics, tracing component failure to effected components, which in turn, reports failure information up to the VSM for acknowledgement and display. This paper will describe the detailed technology and autonomous systems developed, and the integrated multi-module demonstrations conducted. Also, challenges that must be met to fully implement the GAD defined by Gateway will be addressed.
NASA's Space Operations Management Office (SOMO) is working toward a goal of providing an integrated infrastructure of mission and data services for space missions undertaken by NASA enterprises. A significant portion of this effort is focused on reducing the cost of these services. We are interested in the potential of autonomy to reduce operations costs. SOMO services support space missions, but are not part of the mission objectives; therefore the level of acceptable risk is very low. In fact, SOMO could be effective ly prevented from applying autonomy if customers merely perceive it as adding risk to their mission(s). We are interested in this workshop from the standpoint of understanding what can be done to realize the potential cost savings due to autonomy while maintaining acceptable risk and serving the needs of our customers. We would like to present our lessons learned so far in adopting autonomy and automation, which we think will contribute to clarifying the challenges facing the use of such technology. SOMO provides services to a diverse and ambitious set of mission customers. Many of these missions are groundbreaking missions for which communications, data, and other operations requirements sometimes cannot be clearly articulated early in the program. This motivates a need for systems that are robust in the face of unanticipated situations so that customer missions are not unreasonably constrained or impacted by "shortcomings" in SOMO services. One of SOMO's primary goals is to realize a paradigm in which SOMO acts as a service provider to organizations that fly space missions for NASA, other government agencies, and even the commercial sector. These organizations purchase SOMO services "by the pound" as customers. We have to provide systems that are not experiments themselves, but rather stable bases from which to do bold experiments. To this end, SOMO also seeks to work closely with industry to see that robust autonomy technology gets infused into products and services for the space industry and beyond. The potential for application of these technologies spans space-based communications networks (e.g. TDRSS) and ground-based assets including communication and tracking antenna systems, data networks, and control centers. There are several problems that are candidates for the application of autonomy, if it can be made reliable enough, including: antenna control, antenna scheduling, communication link scheduling and operation, navigation, attitude determination, fault detection, isolation, and reconfiguration (for spacecraft or ground assets), and mission-level planning and scheduling. Some attempts have been made to apply autonomy and automation in these areas in the past with varying degrees of success. We will present relevant case histories and the lessons inferred from them. Combining this past experience with anticipated future needs, we can clarify the challenges that must be met in order to realize the benefits of autonomy.
The next-generation human spaceflight vehicle is in a unique position to realize the benefits of more than thirty years of technological advancements since the Space Shuttle was designed. Computer enhancements, the emergence of highly reliable decision-making algorithms, and an emphasis on efficiency make an increased use of autonomous systems highly likely. NASA is in a position to take advantage of these advances and apply them to the human spaceflight environment. One of the key paradigm shifts will be the shift, where appropriate, of monitoring, option development, decision-making, and execution responsibility from humans to an Autonomous Flight Management (AFM) system. As an effort to reduce risk for development of an AFM system, NASA engineers are developing a prototype to prove the utility of previously untested autonomy concepts. This prototype, called SMART (Spacecraft Mission Assessment and Replanning Tool), is a functionally decomposed flight management system with an appropriate level of autonomy for each of its functions. As the development of SMART began, the most important and most often asked question was, How autonomous should an AFM system be? A thorough study of the literature through 2002 surrounding autonomous systems has not yielded a standard method for designing a level of autonomy into either a crewed vehicle or an uncrewed vehicle. The current focus in the literature on defining autonomy is centered on developing IQ tests for built systems. The literature that was analyzed assumes that the goal of all systems is to strive for complete autonomy from human intervention, rather than identifying how autonomous each function within the system should have been. In contrast, the SMART team developed a method for determining the appropriate level of autonomy to be designed into each function within a system. This paper summarizes the development of the Level of Autonomy Assessment Tool and its application to the SMART project.
To achieve NASA’s Artemis program mission objectives a high level of autonomy that is ubiquitous throughout the systems that are being developed will be necessary. The autonomous systems of Artemis will require a distributed autonomy capability, with autonomous systems organized functionally in a hierarchical architecture, where systems at higher levels of the hierarchy have authority over systems at lower levels. The challenge of developing autonomy technologies and Concepts of Operations (ConOps) for Artemis has been undertaken by the NASA Gateway Working Group. This group has developed requirements, architectures, ConOps, and interface control documents (ICDs), in the context of a hierarchical distributed architecture that includes the following: a Vehicle System Manager (VSM) that autonomously manages the entire Gateway; Module System Managers (MSMs) that autonomously manage each module; and System Managers (SMs) that autonomously manage systems within a module (i.e. ECLSS). A substantially high level of autonomy needs to be achieved by each element of the hierarchy (VSM, MSM, SM) to meet requirements for uncrewed operations; this includes conditions that will have minimal and/or delayed ground intervention (i.e. requirements for sustainability for months of operation without crew or ground support). To advance an implementation of this autonomy design (Gateway Autonomy Design – GAD), a collaboration was established between the Autonomous Systems Laboratory (ASL) at NASA Stennis Space Center and Lockheed Martin. The objectives of this partnership were the following: (1) to implement autonomy at the VSM, MSM, and SM levels; (2) to implement communications among a VSM, 2 MSMs, ORION (a visiting vehicle somewhat equivalent to a module) and 1 SM (a power system), and (3) test autonomous operations with representative use cases. A SM backed by a high-fidelity simulation was created to facilitate demonstrations of use cases that originated in a system of a module. Communication between the VSM and MSMs was implemented according to Concepts of Operations and Interface Control Documents (ICDs). Demonstrations were conducted to address nominal and off-nominal operations and multi-module interactions with the VSM. Additionally, user interfaces were created to provide awareness about ongoing processes and results while enhancing the demonstration. Demonstrations included the following use cases: (1) Orion as visiting vehicle registers with VSM; (2) VSM reschedules a module’s timelines when another module’s MSM task fails; and (3) a module’s Power System Manager (PSM) demonstration that included component failure diagnostics, tracing component failure to effected components, which in turn, reports failure information up to the VSM for acknowledgement and display. This paper will describe the detailed technology and autonomous systems developed, and the integrated multi-module demonstrations conducted. Also, challenges that must be met to fully implement the GAD defined by Gateway will be addressed.
NASA’s Moon to Mars architecture is an ambitious roadmap of manned cislunar and deep space exploration. The extensive amount of orbital assets required will place a significant burden on ground-based resources, such as communication networks and operations facilities. Spacecraft autonomy is essential for maintaining a vast number of complex missions beyond Earth orbit. To achieve full autonomy, spacecraft must be able to employ methods of robust maneuver design without an explicit dependence on commands sent from the ground. This level of autonomy is needed not only for stationkeeping, but also for outbound transfers. To address the need of spacecraft maneuver design autonomy, this work investigates the use of neural networks (NNs) in a supervised learning environment. A supervised learning approach for NNs allows for a curated training data set, consisting exclusively of perturbations applied to a desired mission concept of operations (ConOps). The proposed approach allows humans on the ground to design a specific mission ConOps before flight, then employ NNs to fly the mission robustly and autonomously. This investigation numerically tests maneuver autonomy in four highly sensitive regions of flight: orbit raising, translunar injection burns, powered lunar flybys, and invariant manifold insertion burns. These straining cases are contextualized by testing them in a demonstration mission, targeting an Earth-Moon L3 orbit. The study first establishes feasibility by automating impulsive burn maneuvers. However, some guidance algorithms will need more intensive commands, such as inertial pointing and angular rates. To validate this method, NN maneuver autonomy is applied to a finite burn model of the demonstration mission. The use of sequential, mission specific maneuvers provide an appropriate testbed to demonstrate the robustness of a NN trained on feasible perturbed states. Moreover, these scenarios provide preliminary proof-of-concept for fully autonomous missions that execute maneuvers without dependence upon explicit command uplinks. As a result, the technological advancement proposed in this work may significantly ease the strain on ground-based mission operations. This would enable complex and autonomous mission execution in cislunar and deep space regimes, filling a technology gap required to support future manned missions.
Within the last decade, both the U.S. and the Soviet space programs have taken significant preliminary steps in developing technology and systems which are appropriate for the establishment of space stations. The degree of autonomy which will be provided for the station and the role of the crew represents one of the most critical considerations. The present investigation is concerned with a review of the major autonomy issues associated with a permanent, low earth orbit, operational space station. It is shown that both operational effectiveness and crew safety issues require a relatively high degree of space station autonomy. The autonomy level should, for instance, be higher than that of the present space shuttle. Attention is given to various levels of spacecraft autonomy, system integrity, attitude determination and control, navigation and orbit maintenance, system maintenance and resupply, mission support, and implementation of autonomy.
This paper describes the concept of moral hazard as applied to technologies that incorporate automation and autonomy. Moral hazard is said to exist when a party to a transaction feels more comfortable taking undue risks because another party will bear the costs if things go badly. As opposed to regular physical hazards, a moral hazard comes from within a person. In this paper, we reveal two categories of moral hazards related to autonomy. The first category of moral hazard occurs when the owner of the autonomy introduces an autonomous system without accepting the full responsibility for improper operation thereby shifting the risks from one party to another party. This category of moral hazard is similar to moral hazards experienced in other industries and can often be addressed through appropriate policy and establishing liability for irresponsible behavior. The issue becomes more complicated in cases where the operator of the autonomy may not have a full understanding of the system behavior. In the second category of moral hazard, risks are shifted from people to autonomy. In this category, the humans in proximity to the autonomous system begin to trust its behavior. Their behavior may change in that they may believe they are more insulated from harm and subsequently exhibit more risky behavior towards increasingly autonomous technologies. Mitigating this type of moral hazard may require the autonomy to possess certain design features to discourage this type of harmful human behavior so that humans do not suffer needlessly in their interactions with autonomous systems by placing inappropriate trust where that trust is neither warranted nor deserved.
This paper summarizes key findings related to methods for the risk considerations of autonomy software. Existing methods for Verification and Validation (V&V) of autonomy software are summarized and a method for the assurance of autonomy software is suggested and demonstrated on a command execution use case. Risk and reliability are defined in the context of autonomy and an approach for risk assessment of autonomy is presented using an example use case. Key insights regarding areas of uncertainty for autonomy are provided, along with a suggested architecture for the systematic consideration of reliability within the context of a given autonomous planner.
Traditionally, air traffic management services have been provided by air traffic controllers and managers stationed in ground facilities, employed or contracted by the public sector, and supported by automation. These centralized, human-centric air traffic management services do not scale to accommodate increasing demands from conventional and new entrant operations for access to the national airspace system. One transformation that provides much needed scalability is increasing the level of autonomy of air traffic management by enabling edge agents of the system, including vehicles, operators, and third-party service suppliers, to collectively self-manage independently from the centralized service providers and enabling the automation to also take on more independent traffic management responsibility from the human agents. This paper identifies challenges to increasing the level of autonomy of air traffic management services. It describes a framework to enable a systematic identification of these challenges. The framework consists of a functional breakdown of air traffic management services and several dimensions characterizing different autonomy scales. The autonomy dimensions include the automation level between human and machine agents, the locus of control between centralized and distributed edge agents, cognitive activities for autonomous situation awareness and decision making, intelligence levels ranging from skill-based to expertise-based autonomous behavior, and uncertainty levels of the dynamics and environment in which autonomous agents operate. Several challenges are identified and categorized using the different dimensions of the autonomy framework.
Space missions are currently being designed and developed to rely on increased spacecraft autonomy in order to achieve lower cost operations and to support the fleets of planetary spacecraft with limited deep space network resources. In relation to this, metrics are presented which quantitatively define spacecraft autonomy and will be used to: set measurable autonomy goals for future missions; evaluate and compare the benefits between competing automation technologies, and to compare autonomy between missions. The metrics measure the degree to which spacecraft and space mission designs lead to: longer periods of no-track; shorter track periods; reduced ground-orbit communications, and smaller operations workforces. The results of a survey applying these metrics to historic missions and to planned future missions are reported on, illustrating the differences between the autonomy achieved by previous missions and that predicted for future missions.
We offered a first Robotic Autonomy course this summer, located at NASA/Ames' new NASA Research Park, for approximately 30 high school students. In this 7-week course, students worked in ten teams to build then program advanced autonomous robots capable of visual processing and high-speed wireless communication. The course made use of challenge-based curricula, culminating each week with a Wednesday Challenge Day and a Friday Exhibition and Contest Day. Robotic Autonomy provided a comprehensive grounding in elementary robotics, including basic electronics, electronics evaluation, microprocessor programming, real-time control, and robot mechanics and kinematics. Our course then continued the educational process by introducing higher-level perception, action and autonomy topics, including teleoperation, visual servoing, intelligent scheduling and planning and cooperative problem-solving. We were able to deliver such a comprehensive, high-level education in robotic autonomy for two reasons. First, the content resulted from close collaboration between the CMU Robotics Institute and researchers in the Information Sciences and Technology Directorate and various education program/project managers at NASA/Ames. This collaboration produced not only educational content, but will also be focal to the conduct of formative and summative evaluations of the course for further refinement. Second, CMU rapid prototyping skills as well as the PI's low-overhead perception and locomotion research projects enabled design and delivery of affordable robot kits with unprecedented sensory- locomotory capability. Each Trikebot robot was capable of both indoor locomotion and high-speed outdoor motion and was equipped with a high-speed vision system coupled to a low-cost pan/tilt head. As planned, follow the completion of Robotic Autonomy, each student took home an autonomous, competent robot. This robot is the student's to keep, as she explores robotics with an extremely capable tool in the midst of a new community for roboticists. CMU provided undergraduate course credit for this official course, 16-162U, for 13 students, with all other students receiving course credit from National Hispanic University.
The successful operation of unmanned air vehicles requires software with a high degree of autonomy. Only if high level functions can be carried out without human control and intervention, complex missions in a changing and potentially unknown environment can be carried out successfully. Autonomy software is highly mission and safety critical: failures, caused by flaws in the software cannot only jeopardize the mission, but could also endanger human life (e.g., a crash of an UAV in a densely populated area). Due to its large size, high complexity, and use of specialized algorithms (planner, constraint-solver, etc.), autonomy software poses specific challenges for its verification, validation, and certification. -- - we have carried out a survey among researchers aid scientists at NASA to study these issues. In this paper, we will present major results of this study, discussing the broad spectrum. of notions and characteristics of autonomy software and its challenges for design and development. A main focus of this survey was to evaluate verification and validation (V&V) issues and challenges, compared to the development of "traditional" safety-critical software. We will discuss important issues in V&V of autonomous software and advanced V&V tools which can help to mitigate software risks. Results of this survey will help to identify and understand safety concerns in autonomy software and will lead to improved strategies for mitigation of these risks.