Search NASA⌕ Search

SEARCH · Search NASA

Results for “Q learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

Control of matrix converters using machine learning

A method of controlling a matrix converter system is provided. The method includes receiving an operating condition and consulting a trained Q-data structure for reward values associated with respective switching states of the switching matrix for an operating state that corresponds to the operating condition. The Q-data structure is trained using Q-learning to map a reward value predicted for respective switching states to respective discrete operating states. The method further includes sorting the reward values predicted for the respective switching states mapped to the operating state that corresponds to the operating condition, selecting a subset of the set of the mappings as a function of a result of sorting the reward values associated with the switching states of the operating state, evaluating each switching state included in the subset, and selecting an optimal switching state for the operating condition based on a result of evaluating the switching states of the subset.

Chamie, Mahmoud El↗

Deep Reinforcement Learning based Routing in an Air-to-Air Ad-hoc Network

This paper studies the Multiple Sources and Multiple Destinations (MSMD) routing problem in a dynamic Air-to-Air Ad-hoc Network (AAAN). We consider a spectrum limited scenario where multiple links have to share the same frequency channel so that co-channel interference becomes inevitable. As a result, routing decisions and spectrum access are coupled and must be jointly considered. This paper proposes a deep Q-learning based algorithm to find an optimal routing and channel selection strategy that minimizes the end-to-end communication delay. Specifically, under the assumption that only local information is available to every node, the Deep Q-Network (DQN) is trained offline to learn the optimal routing and channel selection strategy. After the trained DQN is implemented in every node, multiple relay nodes can simultaneously determine their next-hop relay and channel selections in real-time. Simulation results demonstrate the efficacy of our proposed algorithm.

AAAN↗

Deep Reinforcement Learning based Routing in an Air-to-Air Ad-hoc Network

This paper studies the Multiple Sources and Multiple Destinations (MSMD) routing problem in a dynamic Air-to-Air Ad-hoc Network (AAAN). We consider a spectrum limited scenario where multiple links have to share the same frequency channel so that co-channel interference becomes inevitable. As a result, routing decisions and spectrum access are coupled and must be jointly considered. This paper proposes a deep Q-learning based algorithm to find an optimal routing and channel selection strategy that minimizes the end-to-end communication delay. Specifically, under the assumption that only local information is available to every node, the Deep Q-Network (DQN) is trained offline to learn the optimal routing and channel selection strategy. After the trained DQN is implemented in every node, multiple relay nodes can simultaneously determine their next-hop relay and channel selections in real-time. Simulation results demonstrate the efficacy of our proposed algorithm.

AAAN↗

On the integration of reinforcement learning and approximate reasoning for control

The author discusses the importance of strengthening the knowledge representation characteristic of reinforcement learning techniques using methods such as approximate reasoning. The ARIC (approximate reasoning-based intelligent control) architecture is an example of such a hybrid approach in which the fuzzy control rules are modified (fine-tuned) using reinforcement learning. ARIC also demonstrates that it is possible to start with an approximately correct control knowledge base and learn to refine this knowledge through further experience. On the other hand, techniques such as the TD (temporal difference) algorithm and Q-learning establish stronger theoretical foundations for their use in adaptive control and also in stability analysis of hybrid reinforcement learning and approximate reasoning-based controllers.

Berenji, Hamid R.↗

Optimal Coordination of Distributed Energy Resources Using Deep Deterministic Policy Gradient

Recent studies showed that reinforcement learning (RL) is a promising approach for coordination and control of distributed energy resources (DER) under uncertainties. Many existing RL approaches, including Q-learning and approximate dynamic programming, are based on lookup table methods, which become inefficient when the problem size is large and infeasible when continuous states and actions are involved. In addition, when modeling battery energy storage system (BESS), the loss of life is not reasonably considered into the decision-making process. This paper proposes an innovative deep RL method for DER coordination considering BESS degradation. The proposed deep RL is designed based on an adaptive actor-critic architecture and employs an off-policy deterministic policy gradient method for determining the dispatch operation that minimizes the operation cost and BESS life loss. Case studies were performed to validate the proposed method and demonstrate the effects of incorporating degradation models into control design.

Das, Avijit↗

Postdisaster Routing of Movable Energy Resources for Enhanced Distribution System Resilience: A Deep Reinforcement Learning-Based Approach

The deployment of movable energy resources (MERs) can be an effective strategy to restore critical loads to enhance power system resilience when no other energy sources are available after the occurrence of an extreme event. Since the optimal locations of MERs following an extreme event are dependent on system operating states (e.g., the loads at each node, on/off status of system branches, and so on), existing analytical and population-based approaches must repeat the entire analysis and calculation when the system operating states change. On the contrary, if deep reinforcement learning (DRL)-based algorithms are sufficiently trained with a wide range of scenarios, they can quickly find optimal or near-optimal locations irrespective of changes in system states. A deep Q-learning-based approach is proposed for optimal MER deployment to enhance power system resilience. MERs can be also utilized to complement other types of resources, if available. The proposed approach operates in two stages after the occurrence of extreme events. In the first stage, the distribution network is represented as a graph, and the network is then reconfigured using tie switches by using Kruskal’s spanning forest search algorithm (KSFSA). To maximize critical load recovery, the optimal or near-optimal locations of MERs are chosen in the second stage. Further, case studies on a 33-node distribution system and a modified IEEE 123-node system demonstrate the effectiveness of the proposed approach for postdisaster routing of MERs.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Short-Term Probabilistic Solar Forecasting via Reinforcement Learning over ECMWF

In this paper, we present an innovative reinforcement learning approach for short-term solar forecasting, leveraging data from the European Centre for Medium-Range Weather Forecasts (ECMWF). The methodology begins with the application of the System Advisor Model (SAM) to transform various ECMWF numerical weather prediction members into predictive photovoltaic power generation. To enhance the precision of deterministic forecasting, we introduce a dynamic model selection algorithm based on Q-learning. This algorithm dynamically identifies and utilizes the most accurate ensemble member for forecasting purposes. Furthermore, we employ a support vector regression surrogate model with a Gaussian distribution to generate probabilistic forecasts, providing a holistic view of solar energy generation uncertainty. To expedite the training process and make it more practical for real-world applications, we integrate a rolling update workflow. This innovative workflow reduces the training period from months to a mere 19 days, making our method highly efficient. Numerical results of the case study show that in comparison to benchmark models, the proposed method improves the deterministic and probabilistic solar forecasting accuracy by up to 40.84% and 48.42%, respectively.

ensemble forecasting↗

SU-D-BRB-05: Quantum Learning for Knowledge-Based Response-Adaptive Radiotherapy

Purpose: There is tremendous excitement in radiotherapy about applying data-driven methods to develop personalized clinical decisions for real-time response-based adaptation. However, classical statistical learning methods lack in terms of efficiency and ability to predict outcomes under conditions of uncertainty and incomplete information. Therefore, we are investigating physics-inspired machine learning approaches by utilizing quantum principles for developing a robust framework to dynamically adapt treatments to individual patient’s characteristics and optimize outcomes. Methods: We studied 88 liver SBRT patients with 35 on non-adaptive and 53 on adaptive protocols. Adaptation was based on liver function using a split-course of 3+2 fractions with a month break. The radiotherapy environment was modeled as a Markov decision process (MDP) of baseline and one month into treatment states. The patient environment was modeled by a 5-variable state represented by patient’s clinical and dosimetric covariates. For comparison of classical and quantum learning methods, decision-making to adapt at one month was considered. The MDP objective was defined by the complication-free tumor control (P{sup +}=TCPx(1-NTCP)). A simple regression model represented state-action mapping. Single bit in classical MDP and a qubit of 2-superimposed states in quantum MDP represented the decision actions. Classical decision selection was done using reinforcement Q-learning and quantum searching was performed using Grover’s algorithm, which applies uniform superposition over possible states and yields quadratic speed-up. Results: Classical/quantum MDPs suggested adaptation (probability amplitude ≥0.5) 79% of the time for splitcourses and 100% for continuous-courses. However, the classical MDP had an average adaptation probability of 0.5±0.22 while the quantum algorithm reached 0.76±0.28. In cases where adaptation failed, classical MDP yielded 0.31±0.26 average amplitude while the quantum approach averaged a more optimistic 0.57±0.4, but with high phase fluctuations. Conclusion: Our results demonstrate that quantum machine learning approaches provide a feasible and promising framework for real-time and sequential clinical decision-making in adaptive radiotherapy.

60 APPLIED LIFE SCIENCES↗

Vehicle to Grid Frequency Regulation Capacity Optimal Scheduling for Battery Swapping Station Using Deep Q-Network

Battery swapping stations (BSSs) are ideal candidates for fast frequency regulation services (FFRS) due to their large battery stock capacity. In addition, BSSs can precharge batteries for customers and the batteries that are not in charging can provide a stable regulation capacity to the market. However, uncertainties, such as ACE signals and the EV per-hour visit counts, introduce stochastic nonlinear dynamics into the operation of a BSS-based FFRS. Currently, there is no quantification method to ensure its optimal economical operation. To close this gap, in this article, we propose a novel deep Q-learning-based FFRS capacity dynamic scheduling strategy. This method can autonomously schedule the hourly regulation capacity in real time to maximize the BSSx0027;s revenue for providing FFRS. Case studies using real-world data verify the efficacy of the proposed work.

25 ENERGY STORAGE↗

K-Spin Hamiltonian for Quantum-Resolvable Markov Decision Processes

The Markov decision process is the mathematical formalization underlying the modern field of reinforcement learning when transition and reward functions are unknown. We derive a pseudo-Boolean cost function that is equivalent to a K-spin Hamiltonian representation of the discrete, finite, discounted Markov decision process with infinite horizon. This K-spin Hamiltonian furnishes a starting point from which to solve for an optimal policy using heuristic quantum algorithms such as adiabatic quantum annealing and the quantum approximate optimization algorithm on near-term quantum hardware. In arguing that the variational minimization of our Hamiltonian is approximately equivalent to the Bellman optimality condition for a prevalent class of environments we establish an interesting analogy with classical field theory. Along with proof-of-concept calculations to corroborate our formulation by simulated and quantum annealing against classical Q-Learning, we analyze the scaling of physical resources required to solve our Hamiltonian on quantum hardware.

Hamiltonian↗

Secure mmWave Spectrum Sharing with Autonomous Beam Scheduling for 5G and Beyond

Spectrum Sharing (SS) has seen a renewed set of initiatives in 5G with the availability of shared and unlicensed spectrum bands that can be used by multiple cellular service providers and private cellular networks. Beam based transmission, instead of the traditional sector based transmission in conjunction with the spectrum agility of the 5G New Radio (NR) has brought new opportunities to optimized sharing of spectrum. Currently in the U.S., a centralized Spectrum Access Server (SAS) is used to co-ordinate spectrum sharing among networks sharing the same spectrum band. However, SAS becomes a focal point for security attacks and a performance bottleneck. In addition, SAS relies on an Environmental Sensor Network (ESN), separate from the 5G network. Without trusted spectral occupancy information, false reporting of spectrum sensing data can create sub-optimal and unfair spectrum usage. This paper summarizes our recent research findings in using a decentralized scheme for multiple networks to securely share spectrum with autonomous beam scheduling : 1) A new stochastic network framework based on Lyapunov Optimization approach is developed to optimize scheduling at the base stations; 2) Game theoretic (GT) approach is used to formulate the distributed scheduler; 3) Another distributed scheduler with Q-learning is presented that utilizes the Reinforcement Learning (RL) approach; 4) The performance and convergence rate of these distributed solutions to use shared and unlicensed spectrum are compared with existing solutions. Conditions under which the performance of these schedulers approach the theoretical upper bound, which is the performance possible with no interference among the operators sharing the spectrum, are presented; 5) The ability of a base station to use its own user equipment as sensors, for optimal spectrum sharing with base stations in other operator networks, is demonstrated to be an effective approach.

5G↗

HackAttack: Game-Theoretic Analysis of Realistic Cyber Conflicts

Game theory is appropriate for studying cyber conflict because it allows for an intelligent and goal-driven adversary. Applications of game theory have led to a number of results regarding optimal attack and defense strategies. However, the overwhelming majority of applications explore overly simplistic games, often ones in which each participant s actions are visible to every other participant. These simplifications strip away the fundamental properties of real cyber conflicts: probabilistic alerting, hidden actions, unknown opponent capabilities. In this paper, we demonstrate that it is possible to analyze a more realistic game, one in which different resources have different weaknesses, players have different exploits, and moves occur in secrecy, but they can be detected. Certainly, more advanced and complex games are possible, but the game presented here is more realistic than any other game we know of in the scientific literature. While optimal strategies can be found for simpler games using calculus, case-by-case analysis, or, for stochastic games, Q-learning, our more complex game is more naturally analyzed using the same methods used to study other complex games, such as checkers and chess. We define a simple evaluation function and employ multi-step searches to create strategies. We show that such scenarios can be analyzed, and find that in cases of extreme uncertainty, it is often better to ignore one s opponent s possible moves. Furthermore, we show that a simple evaluation function in a complex game can lead to interesting and nuanced strategies.

botnets↗

Predictive Data-driven Platform for Subsurface Energy Production

Subsurface energy activities such as unconventional resource recovery, enhanced geothermal energy systems, and geologic carbon storage require fast and reliable methods to account for complex, multiphysical processes in heterogeneous fractured and porous media. Although reservoir simulation is considered the industry standard for simulating these subsurface systems with injection and/or extraction operations, reservoir simulation requires spatio-temporal “Big Data” into the simulation model, which is typically a major challenge during model development and computational phase. In this work, we developed and applied various deep neural network-based approaches to (1) process multiscale image segmentation, (2) generate ensemble members of drainage networks, flow channels, and porous media using deep convolutional generative adversarial network, (3) construct multiple hybrid neural networks such as convolutional LSTM and convolutional neural network-LSTM to develop fast and accurate reduced order models for shale gas extraction, and (4) physics-informed neural network and deep Q-learning for flow and energy production. We hypothesized that physicsbased machine learning/deep learning can overcome the shortcomings of traditional machine learning methods where data-driven models have faltered beyond the data and physical conditions used for training and validation. We improved and developed novel approaches to demonstrate that physics-based ML can allow us to incorporate physical constraints (e.g., scientific domain knowledge) into ML framework. Outcomes of this project will be readily applicable for many energy and national security problems that are particularly defined by multiscale features and network systems.

58 GEOSCIENCES↗

Unifying Temporal and Structural Credit Assignment Problems

Single-agent reinforcement learners in time-extended domains and multi-agent systems share a common dilemma known as the credit assignment problem. Multi-agent systems have the structural credit assignment problem of determining the contributions of a particular agent to a common task. Instead, time-extended single-agent systems have the temporal credit assignment problem of determining the contribution of a particular action to the quality of the full sequence of actions. Traditionally these two problems are considered different and are handled in separate ways. In this article we show how these two forms of the credit assignment problem are equivalent. In this unified frame-work, a single-agent Markov decision process can be broken down into a single-time-step multi-agent process. Furthermore we show that Monte-Carlo estimation or Q-learning (depending on whether the values of resulting actions in the episode are known at the time of learning) are equivalent to different agent utility functions in a multi-agent system. This equivalence shows how an often neglected issue in multi-agent systems is equivalent to a well-known deficiency in multi-time-step learning and lays the basis for solving time-extended multi-agent problems, where both credit assignment problems are present.

Agogino, Adrian K.↗

GraMeR: Gra ph Me ta R einforcement learning for multi-objective influence maximization

Influence maximization (IM) is a combinatorial problem of identifying a subset of seed nodes in a network (graph), which when activated, provide a maximal spread of influence in the network for a given diffusion model and a budget for seed set size. IM has numerous applications such as viral marketing, epidemic control, sensor placement and other network-related tasks. However, its practical uses are limited due to the computational complexity of current algorithms. Recently, deep reinforcement learning has been leveraged to solve IM in order to ease the computational burden. However, there are serious limitations in current approaches, including narrow IM formulation that only consider influence via spread and ignore self-activation, low scalability to large graphs, and lack of generalizability across graph families leading to a large running time for every test network. In this work, we address these limitations through a unique approach that involves: (1) Formulating a generic IM problem as a Markov decision process that handles both intrinsic and influence activations; (2)incorporating generalizability via meta-learning across graph families. There are previous works that combine deep reinforcement learning with graph neural network, but this work solves a more realistic IM problem and incorporates generalizability across graphs via meta reinforcement learning. Extensive experiments are carried out in various standard networks to validate performance of the proposed Graph Meta Reinforcement learning (GraMeR) framework. Finally, the results indicate that GraMeR is multiple orders faster and generic than conventional approaches when applied on small to medium scale graphs.

97 MATHEMATICS AND COMPUTING↗

Fusing TensorFlow with building energy simulation for intelligent energy management in smart cities

Buildings account for 35% of the global final energy demand. Efficiency improvements and advanced control strategies have a significant impact in the reduction of energy costs and CO2 emissions. Building energy simulation is widely used to help planners, contractors, and building owners analyse diverse options regarding the planning and management of energy consumption in buildings. Furthermore, recent advances in data processing and computing have led to the development of sophisticated machine learning algorithms that can learn from large datasets, e.g., sensor data from buildings, and use them to develop building-specific adaptive and automatic energy controllers. Control algorithms, such as deep reinforcement learning can tune themselves, are model-free, and economical to implement. In this paper, we introduce an integrated simulation environment that combines CitySim, a fast building energy simulator, and TensorFlow, a platform for efficient implementation of advanced machine learning algorithms. The integration is achieved via Keras - an API for TensorFlow - and a set of text and csv files for data transfer between the applications. This new environment will allow researchers to investigate novel learning control algorithms, and demonstrate their robustness and potential for diverse applications in the built environment. We present two case studies for energy savings and demand response, respectively.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Physical Interpretation of Early Battery Life Prediction Models

Early battery life prediction models are most useful for R&D if they help us understand the early changes in battery electrochemical response that correspond with long-term degradation and failure. Linear regression models such as Fused lasso and Partial Least Squares can fit coefficients directly to high-dimensional electrochemical data like capacity-voltage and ΔV–state-of-charge, i.e., Q(V) and ΔV(SOC) curves, learning coefficients that can be physically interpreted. We leverage the ISU-ILCC battery aging data set to learn high-dimensional coefficients for early battery life prediction from traditional slow-rate capacity check data, demonstrating learning on Q(V), d Q· d V −1 , and ΔV(SOC) curves. A thorough study on the dependence of coefficient values on train/test size and data preprocessing methods is made, demonstrating the reliability of high-dimensional regression approaches unless very small amounts of data are used for model training. For this data set, coefficients from Q(V) and d Q· d V −1 models highlight changes in electrode stoichiometry due to lithium loss, while ΔV(SOC) coefficients highlight changes in positive electrode diffusivity due to particle cracking as well as electrode stoichiometry shifts. By directly interpreting the coefficients of a regression model, we make physical insights into battery degradation mechanisms without requiring the assumptions of traditional battery data analysis methods.

25 ENERGY STORAGE↗

Joint Spectrum Access and Power Control in Air-Air Communications - A Deep Reinforcement Learning Based Approach

This paper considers the dynamic spectrum access and power control problem in a single-hop point-to-point Air-Air Communication Network (AACN). Due to spectrum scarcity, we assume the number of Aircraft-to-Aircraft (A2A) communication links is greater than that of the available channels, such that some communication links need to share the same channel, causing co-channel interference. We formulate the joint channel selection and power control optimization problem to maximize the Weighted Sum Spectral Efficiency (WSSE). A distributed and dynamic deep Q learning-based algorithm is proposed to find the optimal solution. Specifically, we design two different policies that are trained by conducting a trial-and-error scheme. Each communication link can achieve the optimal policy by exploiting the local information from its neighbors, and this distributive approach make it scalable to large networks. Finally, our experimental results demonstrate the effectiveness of the proposed solution in various AACN scenarios.

Zhe Wang↗