Utilizing Reinforcement Learning to Learn Safe Conflict Resolution
Explore the source record for details and available documents.
Brandon Waddell↗
SEARCH · Search NASA
Results for “safe reinforcement learning”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
SoMoGym: A Toolkit for Developing and Evaluating Controllers and Reinforcement Learning Algorithms for Soft Robots
Soft robotsoffer a host of benefits over traditional rigid robots, including inherent compliance that lets them passively adapt to variable environments and operate safely around humans and fragile objects. However, that same compliance makes it hard to use model-based methods in planning tasks requiring high precision or complex actuation sequences. Reinforcement learning (RL) can potentially find effective control policies, but training RL using physical soft robots is often infeasible, and training using simulations has had a high barrier to adoption. To accelerate research in control and RL for soft robotic systems, we introduce SoMoGym ( So ft Mo tion Gym ), a software toolkit that facilitates training and evaluating controllers for continuum robots. SoMoGym provides a set of benchmark tasks in which soft robots interact with various objects and environments. It allows evaluation of performance on these tasks for controllers of interest, and enables the use of RL to generate new controllers. Custom environments and robots can likewise be added easily. We provide and evaluate baseline RL policies for each of the benchmark tasks. These results show that SoMoGym enables the use of RL for continuum robots, a class of robots not covered by existing benchmarks, giving them the capability to autonomously solve tasks that were previously unattainable.
Moritz A. Graule↗
Joint Communication Resource Allocation and Velocity Selection in Urban Air Mobility via Multi-agent Reinforcement Learning
With traffic congestion problems becoming more severe in urban areas, the National Aeronautics and Space Administration promotes the Urban Air Mobility (UAM) concept, which envisages a safe and efficient air transportation system. However, the increased communication demands in UAM can exacerbate the spectrum scarcity. Therefore, a new communication resource allocation solution is necessary. In this paper, we focus on uplink UAM communications, where multiple aerial vehicles (AV) perform cargo/passenger delivery tasks. With predefined flight paths, AVs make decisions on communication resource allocation and velocity selection to complete their missions under safety constraints. Accordingly, we formulate a joint optimization problem to minimize the weighted sum of the total travel time and communication outage time. We first model the optimization problem as a Markov game and propose a multi-agent reinforcement learning based solution. Simulation results corroborate the effectiveness of the proposed solution.
Ruixuan Han↗
Using Reinforcement Learning to Design Missed Thrust Resilient Trajectories
From ion thrusters to solar sails, spacecraft continue to adopt new and more efficient forms of propulsion. As these low-thrust propulsion meth- ods have become more prevalent, new challenges have arisen. Depending on the mission, low-thrust propulsion elements may need to thrust con- tinuously for days/months. During these thrusting periods, external fac- tors, such as a micro-meteoroid impact or a software glitch, may cause the spacecraft to prematurely cease its thrust stage. Half of all deep space missions enter a safe mode where they cannot thrust every four months. These missed thrust events can result in the complete loss of a space- craft for time-dependent trajectories like planetary rendezvous. This paper demonstrates how neural networks, trained using reinforcement learning, can autonomously correct for missed thrust events during an interplanetary trajectory.
Laipert, Frank E.↗
Dynamic Spectrum Allocation in Urban Air Transportation System via Deep Reinforcement Learning
The emerging concepts of Urban Air Mobility (UAM) and Advanced Air Mobility (AAM) open a new paradigm for urban air transportation. A big challenge is that these new aerial vehicles will quickly saturate the already crowded aviation spectrum, which is an essential resource to ensure reliable communications for safe operations. In this paper, we consider an air transportation system where multiple aerial vehicles are operated to transport passengers or cargo from different sources to destinations along their pre-defined paths. During the flight, the minimum communication Quality of Service (QoS) requirement must be achieved to ensure flight safety. Our objective is to minimize the average mission completion time by jointly optimizing the velocity selection and spectrum allocation for all aerial vehicles. We formulate the optimization problem as a multi-stage Markov Decision Process (MDP) where the optimization variables are coupled together. A multi-agent Deep Reinforcement Learning (DRL) based solution is proposed where Value Decomposition Networks (VDN) algorithm is utilized to take discrete actions. Additionally, we propose a heuristic greedy algorithm as a baseline solution. Simulation results show that our learning based solution outperforms the heuristic greedy algorithm and another Orthogonal Multiple Access (OMA) solution in minimizing the mission completion time.
Ruixuan Han↗
Dynamic Spectrum Allocation in Urban Air Transportation System via Deep Reinforcement Learning
The emerging concepts of Urban Air Mobility (UAM) and Advanced Air Mobility (AAM) open a new paradigm for urban air transportation. A big challenge is that these new aerial vehicles will quickly saturate the already crowded aviation spectrum, which is an essential resource to ensure reliable communications for safe operations. In this paper, we consider an air transportation system where multiple aerial vehicles are operated to transport passengers or cargo from different sources to destinations along their pre-defined paths. During the flight, the minimum communication Quality of Service (QoS) requirement must be achieved to ensure flight safety. Our objective is to minimize the average mission completion time by jointly optimizing the velocity selection and spectrum allocation for all aerial vehicles. We formulate the optimization problem as a multi-stage Markov Decision Process (MDP) where the optimization variables are coupled together. A multi-agent Deep Reinforcement Learning (DRL) based solution is proposed where Value Decomposition Networks (VDN) algorithm is utilized to take discrete actions. Additionally, we propose a heuristic greedy algorithm as a baseline solution. Simulation results show that our learning based solution outperforms the heuristic greedy algorithm and another Orthogonal Multiple Access (OMA) solution in minimizing the mission completion time.
Ruixuan Han↗
Initial Approach to Collect Small Unmanned Aircraft System Off-Nominal Operational Situations Data
NASA is developing the Unmanned Aircraft System Traffic Management research platform to safely integrate small unmanned aircraft operations in large-scale at low-altitudes. As a part of this effort, small unmanned aircraft system off-nominal operational situations data collection process has been developed to take lessons learned and to reinforce operational compliance. In this paper, descriptions of variables used for digital data collection and an online report form for collection of observational data from the operators (contextual data) are provided. They are used to collect off-nominal data from the Unmanned Aircraft System Traffic Management National Campaign in 2017. The digital data show that 2 out of 118 campaign operations (1.7%) encountered loss of navigation. Since the campaign aircraft used Global Positioning System for navigation, it is likely that unobstructed view of the sky at the campaign locations contributed to this small number. Also, 4 out of 47 operations (8.5%) encountered loss of communications. A relatively short distance between ground control system and aircraft, ranging from 2300 feet to 4200 feet, likely contributed to this small number. There was no data to identify the loss of communications condition, aircraft received signal strength, for the remaining 71 operations suggesting that some operators may not be monitoring unmanned aircraft communications system performance or monitoring it with different parameters. For the contextual data, due to the low number of total reports during the campaign, no significant trends emerged. This is an initial attempt to collect contextual data from small unmanned aircraft operators about off-nominal situations, and changes will be made to the future data collection to improve the amount and quality of the information.
unmanned aviation systems traffic management (UTM)↗
Initial Approach to Collect Small Unmanned Aircraft System Off-Nominal Operational Situations Data
NASA is developing the Unmanned Aircraft System Traffic Management research platform to safely integrate small unmanned aircraft operations in large-scale at low-altitudes. As a part of this effort, small unmanned aircraft system off-nominal operational situations data collection process has been developed to take lessons learned and to reinforce operational compliance. In this paper, descriptions of variables used for digital data collection and an online report form for collection of observational data from the operators (contextual data) are provided. They are used to collect off-nominal data from the Unmanned Aircraft System Traffic Management National Campaign in 2017. The digital data show that 2 out of 118 campaign operations (1.7%) encountered loss of navigation. Since the campaign aircraft used Global Positioning System for navigation, it is likely that unobstructed view of the sky at the campaign locations contributed to this small number. Also, 4 out of 47 operations (8.5%) encountered loss of communications. A relatively short distance between ground control system and aircraft, ranging from 2300 feet to 4200 feet, likely contributed to this small number. There was no data to identify the loss of communications condition, aircraft received signal strength, for the remaining 71 operations suggesting that some operators may not be monitoring unmanned aircraft communications system performance or monitoring it with different parameters. For the contextual data, due to the low number of total reports during the campaign, no significant trends emerged. This is an initial attempt to collect contextual data from small unmanned aircraft operators about off-nominal situations, and changes will be made to the future data collection to improve the amount and quality of the information.
Jung, Jaewoo↗
Hazard Contribution Modes of Machine Learning Components
Amongst the essential steps to be taken towards developing and deploying safe systems with embedded learning-enabled components (LECs) i.e., software components that use ma- chine learning (ML)—are to analyze and understand the con- tribution of the constituent LECs to safety, and to assure that those contributions have been appropriately managed. This paper addresses both steps by, first, introducing the notion of hazard contribution modes (HCMs) a categorization of the ways in which the ML elements of LECs can contribute to hazardous system states; and, second, describing how argumentation patterns can capture the reasoning that can be used to assure HCM mitigation. Our framework is generic in the sense that the categories of HCMs developed i) can admit different learning schemes, i.e., supervised, unsupervised, and reinforcement learning, and ii) are not dependent on the type of system in which the LECs are embedded, i.e., both cyber and cyber-physical systems. One of the goals of this work is to serve a starting point for systematizing L analysis towards eventually automating it in a tool.
Smith, Colin↗