Search NASA⌕ Search

SEARCH · Search NASA

Results for “Learning control”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24

A synthetic building operation dataset

Abstract This paper presents a synthetic building operation dataset which includes HVAC, lighting, miscellaneous electric loads (MELs) system operating conditions, occupant counts, environmental parameters, end-use and whole-building energy consumptions at 10-minute intervals. The data is created with 1395 annual simulations using the U.S. DOE detailed medium-sized reference office building, and 30 years’ historical weather data in three typical climates including Miami, San Francisco, and Chicago. Three energy efficiency levels of the building and systems are considered. Assumptions regarding occupant movements, occupants’ diverse temperature preferences, lighting, and MELs are adopted to reflect realistic building operations. A semantic building metadata schema - BRICK, is used to store the building metadata. The dataset is saved in a 1.2 TB of compressed HDF5 file. This dataset can be used in various applications, including building energy and load shape benchmarking, energy model calibration, evaluation of occupant and weather variability and their influences on building performance, algorithm development and testing for thermal and energy load prediction, model predictive control, policy development for reinforcement learning based building controls.

24 POWER TRANSMISSION AND DISTRIBUTION↗

AlphaBuilding - Synthetic Buildings Operation Dataset

This is a synthetic building operation dataset which includes HVAC, lighting, miscellaneous electric loads (MELs) system operating conditions, occupant counts, environmental parameters, end-use and whole-building energy consumptions at 10-minute intervals. The data is created with 1395 annual simulations using the U.S. DOE detailed medium-sized reference office building, and 30 years' historical weather data in three typical climates including Miami, San Francisco, and Chicago. Three energy efficiency levels of the building and systems are considered. Assumptions regarding occupant movements, occupants' diverse temperature preferences, lighting, and MELs are adopted to reflect realistic building operations. A semantic building metadata schema - BRICK, is used to store the building metadata. The dataset is saved in a 1.2 TB of compressed HDF5 file. This dataset can be used in various applications, including building energy and load shape benchmarking, energy model calibration, evaluation of occupant and weather variability and their influences on building performance, algorithm development and testing for thermal and energy load prediction, model predictive control, policy development for reinforcement learning based building controls.

AlphaBuilding↗

Feedback Implementation of Zermelo's Optimal Control by Sugeno Approximation

This paper proposes an approach to implement optimal control laws of nonlinear systems in real time. Our methodology does not require solving two-point boundary value problems online and may not require it off-line either. The optimal control law is learned using the original Sugeno controller (OSC) from a family of optimal trajectories. We compare the trajectories generated by the OSC and the trajectories yielded by the optimal feedback control law when applied to Zermelo's ship steering problem.

Clifton, C.↗

A Real-Time Big Data Control-Theoretical Framework for Cyber-Physical-Human Systems [Book Chapter]

It is known that cyber-physical-human systems naturally arise from interdependent infrastructure systems and smart connected communities. Such applications require ubiquitous information sensing and processing, intelligent machine-to-machine communication for a seamless coordination, as well as intelligent interactions between humans and machines. This chapter presents a control-theoretical framework to model heterogeneous physical dynamic systems, information and communication, as well as cooperative controls and/or distributed optimization of such interconnected systems. It is shown that efficient analytical and computational algorithms can be modularly designed and hierarchically implemented to operate and optimize cyber-physical-human systems, first to quantify individually the input–output relationship of nonlinear dynamic behaviors of every physical subsystem, then to coordinate locally both cyber-physical interactions of neighboring agents as well as physical-human interactions, and finally to dynamically model and optimize the overall networked system. The hierarchical structure makes the overall optimization and control problem scalable and solvable. Moreover, the three levels integrate individual designs and optimization, distributed cooperative optimization, and decision-making through real-time, data-driven, model-based learning and control. Specifically, one of the contributions of the chapter is to demonstrate how the combination of dissipativity theory and cooperative control serves as a natural framework and promising tools to analyze, optimize, and control such large-scale system. Application to digital grid is investigated as an illustrative example.

97 MATHEMATICS AND COMPUTING↗

Predicting disruptive instabilities in controlled fusion plasmas through deep learning

Nuclear fusion power delivered by magnetic-confinement tokamak reactors holds the promise of sustainable and clean energy1. The avoidance of large-scale plasma instabilities called disruptions within these reactors is one of the most pressing challenges, because disruptions can halt power production and damage key components. Disruptions are particularly harmful for large burning-plasma systems such as the multibillion-dollar International Thermonuclear Experimental Reactor (ITER) project currently under construction, which aims to be the first reactor that produces more power from fusion than is injected to heat the plasma. Here we present a method based on deep learning for forecasting disruptions. Our method extends considerably the capabilities of previous strategies such as first-principles-based and classical machine-learning approaches. In particular, it delivers reliable predictions for machines other than the one on which it was trained—a crucial requirement for future large reactors that cannot afford training disruptions. Our approach takes advantage of high-dimensional training data to boost predictive performance while also engaging supercomputing resources at the largest scale to improve accuracy and speed. Trained on experimental data from the largest tokamaks in the United States (DIII-D) and the world (Joint European Torus, JET), our method can also be applied to specific tasks such as prediction with long warning times: this opens up the possibility of moving from passive disruption prediction to active reactor control and optimization. These initial results illustrate the potential for deep learning to accelerate progress in fusion-energy science and, more generally, in the understanding and prediction of complex physical systems.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Integrating Overhead Camera Imagery with Reinforcement Learning to Improve Fuel Economy Through Adaptive Traffic Control

Overhead cameras have been deployed for some time as a sensor for traffic control, with many advantages over conventional inductive loop sensors in terms of robustness, ease of installation, and simplicity with respect to maintenance. We explore the utility of such imagery (in particular, fisheye lens cameras) for intelligent traffic control in real-time using reinforcement learning, with a goal of improving fuel economy for the overall intersection and grid of intersections. We summarize our initial work in data collection and approach to ascertain the ability to visually sense vehicular fuel consumption using real-world data. We then describe our deep reinforcement learning topology to test adaptive control informed by the camera capabilities. Next, we tested the proof-of-concept through a variety of simulation scenarios using the Simulation of Urban Mobility (SUMO) traffic simulator, which included different estimates on vehicle classifications, different traffic distributions, and different traffic densities. Six different control policies were tested ranging from simple timing, to heuristic visual-based methods, to actual policies learned via the reinforcement learning approach. Our results demonstrate the feasibility of the approach for controlling traffic lights for better fuel efficiency based on visual vehicle estimates from commercial, fisheye lens cameras.

Karnowski, Thomas↗

Multi-Agent Deep Reinforcement Learning for Realistic Distribution System Voltage Control Using PV Inverters

Over the last few decades, the deployment of distributed solar photovoltaic (PV) systems has increased consistently. High PV penetration could cause adverse effects on the grid, such as voltage violations. This paper proposes a new distributed soft actor-critic based multi-agent deep reinforcement learning (SAC-MADRL) control solution to minimize the PV real power curtailment while keeping the grid voltage in an acceptable range. New reward functions have been designed to coordinate different agents during the learning process, yielding improved convergence. Comparison results with other control methods on a real feeder in western Colorado U.S. with 80% penetration of PVs demonstrate that the proposed method has better capability of effectively regulating voltage while minimizing the PV real power curtailment.

distribution system↗

Deep Reinforcement Learning for Autonomous Water Heater Control

Electric water heaters represent 14% of the electricity consumption in residential buildings. An average household in the United States (U.S.) spends about USD 400–600 (0.45 ¢/L–0.68 ¢/L) on water heating every year. In this context, water heaters are often considered as a valuable asset for Demand Response (DR) and building energy management system (BEMS) applications. To this end, this study proposes a model-free deep reinforcement learning (RL) approach that aims to minimize the electricity cost of a water heater under a time-of-use (TOU) electricity pricing policy by only using standard DR commands. In this approach, a set of RL agents, with different look ahead periods, were trained using the deep Q-networks (DQN) algorithm and their performance was tested on an unseen pair of price and hot water usage profiles. The testing results showed that the RL agents can help save electricity cost in the range of 19% to 35% compared to the baseline operation without causing any discomfort to end users. Additionally, the RL agents outperformed rule-based and model predictive control (MPC)-based controllers and achieved comparable performance to optimization-based control.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

A Data Science and Machine Learning Platform Supporting Large Particle Accelerator Control and Diagnostics Applications Final Report: SBIR Initial Phase II DE-SC0022583

The Machine Learning Data Platform (MLDP) is a product providing full-stack support for data science, Machine Learning, and Artificial Intelligence (ML/AI) applications at particle accelerator and large experimental physics facilities. It supports ML/AI applications from front-end, high-speed acquisition of heterogeneous, time-series data, through data archiving and management, to back-end analysis. The MLDP embodies a “data-science ready” platform for data analysis and ML/AI applications in diagnosis, modelling, control, and optimization of these facilities. It provides data scientists and applications a consistent, datacentric interface to archive data standardizing implementation and deployment of ML/AI algorithms to different operations configurations within the same facility, or between facilities. Being an open-source, public-domain project, the MLDP is intended for broadest possible impact by increasing accessibility and minimizing the required expertise for installation and operation. The MLDP can also be deployed at user facilities for experimental data collection, archiving, and analysis. It is capable of acquisition and archiving of heterogeneous data from experimental equipment (e.g., images, arrays, structures, etc.) along with system hardware configurations (e.g., scalars, tables), control system process variables, and any metadata required for provenance. Thus, the MLDP can manage experimental data through its entire lifecycle, from acquisition and archiving, through analysis and investigation, to release and final publication.

43 PARTICLE ACCELERATORS↗

Deep Reinforcement Learning Based Smart Water Heater Control for Reducing Electricity Consumption and Carbon Emission

Water heating is the third largest electricity consumer in U.S. households, after space heating and cooling. Thus, water heaters represent a significant potential for reducing electricity consumption and associated CO2 emissions of residential buildings. To this end, this study proposes a model-free deep reinforcement learning (RL) approach that aims to minimize the electricity consumption and the CO2 emissions of a heat pump water heater without affecting user comfort. In this approach, a set of RL agents focusing on either electricity saving or emission reduction, with different look ahead periods, were trained using the deep Q-networks (DQN) algorithm and their performance was tested on different hot water usage and Marginal Operating Emissions Rate (MOER) profiles. The testing results showed that the RL agents that focus on electricity saving can save electricity in the range of 12–22% by operating the water heater with maximum heat pump efficiency and minimum electric element utilization. On the other hand, the RL agents that focus on emission reduction reduced emissions in the range of 18–37% by making use of the variable MOER values. These RL agents used the heat pump and/or an element when the MOER values are low due to the availability of renewable energy sources (e.g., solar and wind) and mostly avoided the periods of carbon-intensive periods. Overall, these results showed that the proposed RL approach can help minimize the electricity consumption and the CO2 emissions of a heat pump water heater without having any prior knowledge about the device.

Amasyali, Kadir↗

From Sim to Real: A Pipeline for Training and Deploying Traffic Smoothing Cruise Controllers

Designing and validating controllers for connected and automated vehicles to enhance traffic flow presents significant challenges, from the complexity of replicating real-world stop-and-go traffic dynamics in simulation, to the intricacies involved in transitioning from simulation to actual deployment. In this work, we present a full pipeline from data collection to controller deployment. Specifically, we collect 772 km of driving data from the I-24 in Tennessee, and use it to build a one-lane simulator, placing simulated vehicles behind real-world trajectories. Using policy-gradient methods with an asymmetric critic, we improve fuel efficiency by over 10% when simulating congested scenarios. Our comprehensive approach includes reinforcement learning for controller training, software verification, hardware validation and setup, and navigating various sim-to-real challenges. Furthermore, we analyze the controller's behavior and wave-smoothing properties, and deploy it on four Toyota Rav4’s in a real-world validation experiment on the I-24. Lastly, we release the driving dataset, the simulator and the trained controller, to enable future benchmarking and controller design.

42 ENGINEERING↗

Double Deep Q-Networks for Optimizing Electricity Cost of a Water Heater

Electric water heaters represent 14% of the electricity consumption in the residential buildings and the cost associated with domestic water heating account for a good portion of the household expenses in the United States. In this context, intelligent control of water heaters gained a lot of research attention. In recent years, a significant number of intelligent water heater controllers, with various methods and intended uses, have been proposed. However, existing studies are mostly model-based approaches that require an accurate modelling of the water heater. Towards addressing this research gap, this paper presents a model-free reinforcement learning-based controller for a day-ahead price market. The controller aims to minimize the cost of domestic water heating while maintaining the user comfort. The results showed that the developed controller can help save energy cost while maintaining the temperatures within the desired comfort band.

Amasyali, Kadir↗

Aircraft Control Using Engine Thrust: A History of Learning TOC Real-Time

A history of learning the operation of Throttles Only Control (TOC) to control an aircraft in real time using engine thrust is shown. The topics include: 1) Past TOC Accidents/Incidents; 2) 1972: DC-10 American Airlines; 3) May 1974: USAF B-52H; 4) April 1975: USAF C-5A; 5) April 1975: USAF C-5A; 6) 1981: USAF B-52G; 7) August 1985: JAL 123 B-747; 8) JAL 123 Survivor Story; 9) JAL 123 Investigation Findings; 10) July 1989: UAL 232 DC-10; 11) UAL 232 DC-10; 12) Eastwind 517 B-737; 13) November 2003: DHL A-300; 14) Historically, TOC has saved lives; 15) Automated Throttles-Only Control; 16) PCA Project; 17) Propulsion-Controlled Aircraft; 18) MD-11 PCA System and Flight Test Envelope; 19) MD-11 Simulation, PCA ILS-Soupled Landing Dispersion; 20) Throttles-Only Pitch and Roll Control Power; 21) PCA in Commercial Fleet; 22) Fall 2005: PCAR Project; 23) PCAR Background - TOC; and 24) PCAR Background - TOC.

Cole, Jennifer H.↗

Global-Local Policy Search and its Application in Grid-Interactive Building Control

As the buildings sector represents over 70% of the total U.S. electricity consumption, it offers a great amount of untapped demand-side resources to tackle many critical grid-side problems and improve the overall energy system's efficiency. To help make buildings grid-interactive, this paper proposes a global-local policy search method to train a reinforcement learning (RL) based controller which optimizes building operation during both normal hours and demand response (DR) events. Experiments on a simulated five-zone commercial building demonstrate that by adding a local fine-tuning stage to the evolution strategy policy training process, the control costs can be further reduced by 7.55% in unseen testing scenarios. Baseline comparison also indicates that the learned RL controller outperforms a pragmatic linear model predictive controller (MPC), while not requiring intensive online computation.

demand response↗

Machine Learning-Based Classification of Lignocellulosic Biomass from Pyrolysis-Molecular Beam Mass Spectrometry Data

High-throughput analysis of biomass is necessary to ensure consistent and uniform feedstocks for agricultural and bioenergy applications and is needed to inform genomics and systems biology models. Pyrolysis followed by mass spectrometry such as molecular beam mass spectrometry (py-MBMS) analyses are becoming increasingly popular for the rapid analysis of biomass cell wall composition and typically require the use of different data analysis tools depending on the need and application. Here, the authors report the py-MBMS analysis of several types of lignocellulosic biomass to gain an understanding of spectral patterns and variation with associated biomass composition and use machine learning approaches to classify, differentiate, and predict biomass types on the basis of py-MBMS spectra. Py-MBMS spectra were also corrected for instrumental variance using generalized linear modeling (GLM) based on the use of select ions relative abundances as spike-in controls. Machine learning classification algorithms e.g., random forest, k-nearest neighbor, decision tree, Gaussian Naïve Bayes, gradient boosting, and multilayer perceptron classifiers were used. The k-nearest neighbors (k-NN) classifier generally performed the best for classifications using raw spectral data, and the decision tree classifier performed the worst. After normalization of spectra to account for instrumental variance, all the classifiers had comparable and generally acceptable performance for predicting the biomass types, although the k-NN and decision tree classifiers were not as accurate for prediction of specific sample types. Gaussian Naïve Bayes (GNB) and extreme gradient boosting (XGB) classifiers performed better than the k-NN and the decision tree classifiers for the prediction of biomass mixtures. The data analysis workflow reported here could be applied and extended for comparison of biomass samples of varying types, species, phenotypes, and/or genotypes or subjected to different treatments, environments, etc. to further elucidate the sources of spectral variance, patterns, and to infer compositional information based on spectral analysis, particularly for analysis of data without a priori knowledge of the feedstock composition or identity.

59 BASIC BIOLOGICAL SCIENCES↗

CRCNS US-France Research Proposal: Collaborative Research: Encoding reward expectation in Drosophilia

The fruit fly Drosophila melanogaster has been a valuable model for investigating the genetic and neural bases that underlie learning and memory. Early and most current studies use basic behavior conditioning protocols to study learning in controlled laboratory settings. More recently, the ability to transgenically manipulate many of the brain neurons in the fruit fly with exquisite specificity, and the recent knowledge of the synaptic ‘connectome’ of the fruit fly brain, makes these animals almost unique as a comprehensive model for studies of learning, memory and motivated behavior. In fact, the connectome has revealed many types of new connections that had until now been overlooked. Within this context, the thesis of this proposal is that studies of learning and memory will be greatly enhanced by using more sophisticated means for evaluating memory representations, such as have been developed in vertebrates, and combining those studies with information from the connectome guided by computational modelling. We propose to push beyond the boundaries of existing conditioning protocols for fruit flies to investigate more complex memory representations. In particular, we will investigate the function of reinforcement pathways in relation to the absence of expected reinforcement. More specifically, we propose a series of experiments designed to investigate the memory representations in fruit flies when an expected consequence of a Conditioned Stimulus (CS) fails to occur. Although studies have evaluated how this failure can establish extinction memory for the CS, our studies will go beyond studying extinction. Specifically, we predict that in Drosophila when a CS is associated with a failed expectation of an appetitive food reinforcement it will acquire aversive value, and vice versa for a failed expectation of an aversive reinforcer. We combine these studies with manipulations of reinforcement pathways in the CNS inspired from the connectome, iteratively knitted in with established computational models. Intellectual Merit: The concept of reinforcement expectation and incentive contrast have been influential in the development of studies of associative learning in mammals. These questions are particularly challenging to answer in vertebrates because they require exquisite cellular, temporal, and genetic specificity of experimental manipulations. The recent development of work with identified neurons and their connectomes makes the larval and adult fly brains ripe as models for pushing our understanding of neural bases for these higher- order conditioning phenomena. Broader Impacts: Public health: These analyses and the conceptual framework of prediction error processing underlying them have a profound impact on our understanding of reinforcement-related behavior in humans, including monetary rewards and the mnemonic consequences of traumatic experiences, and for pathologies of the dopamine reinforcement system. Educational: This project will provide interdisciplinary training for postdoctoral researchers, Ph.D. and undergraduate students. The PIs will act as co-supervisors or mentors of students working in the different labs via face-to-face and internet-based technologies. We will also work with ASU’s award-winning Ask- A-Biologist program. This is an online science program designed to enrich the learning experiences of students of all ages and to provide classroom material for use by K-12 teachers. We will develop an extension of a game developed under a prior NSF award, and the new game will include modules to teach K-12 students about how insects learn. We will also integrate into the AAB site a program developed by a collaborator (B Gerber) at the Leibniz Institut für Neurobiologie, Magdeburg, and now in use in schools in Germany, to teach K-12 students how to train animals using the fruit fly larval learning paradigm. Underrepresented groups: All PIs will work with their university offices of Academic Diversity and Equal Opportunity for reaching underrepresented students.

59 BASIC BIOLOGICAL SCIENCES↗

Predicting plug loads with occupant count data through a deep learning approach

Predictive control has gained increasing attention for its ability to reduce energy consumption and improve occupant comfort in buildings. The plug loads prediction is a key component for the predictive building controls, as plug loads is a major source of internal heat gains in buildings. This study proposed a novel method to apply the Long-Short-Term-Memory (LSTM) Network, a special form of Recurrent Neural Network, to predict plug loads. The occupant count and the time have been confirmed to drive the plug load profile and thus selected as the features for the plug load prediction. The LSTM network was trained and tested with ground truth occupant count data collected from a real office building in Berkeley, California. Results from the LSTM network markedly improve the prediction accuracy compared with traditional linear regression methods and the classical Artificial Neural Network. 95% of 1-hour predictions from LSTM network are within ±1 kW of the actual plug loads, given the average plug loads during the office hour is 8.6 kW. The CV(RMSE) of the predicted plug load is 11% for the next hour, and 20% for the next eight hours. Lastly, we compared four prediction approaches with the office building we monitored: LSTM vs. ARIMA, with occupant counts vs. without occupant counts. It was found, the prediction error of the LSTM approach is around 4% less than the ARIMA approach. Using occupant counts as an exogenous input could further reduce the prediction error by 5%-6%. The findings of this paper could shed light on the plug load prediction for building control optimizations such as model-predictive control.

Occupant count↗

Lessons Learned and Flight Results from the F15 Intelligent Flight Control System Project

A viewgraph presentation on the lessons learned and flight results from the F15 Intelligent Flight Control System (IFCS) project is shown. The topics include: 1) F-15 IFCS Project Goals; 2) Motivation; 3) IFCS Approach; 4) NASA F-15 #837 Aircraft Description; 5) Flight Envelope; 6) Limited Authority System; 7) NN Floating Limiter; 8) Flight Experiment; 9) Adaptation Goals; 10) Handling Qualities Performance Metric; 11) Project Phases; 12) Indirect Adaptive Control Architecture; 13) Indirect Adaptive Experience and Lessons Learned; 14) Gen II Direct Adaptive Control Architecture; 15) Current Status; 16) Effect of Canard Multiplier; 17) Simulated Canard Failure Stab Open Loop; 18) Canard Multiplier Effect Closed Loop Freq. Resp.; 19) Simulated Canard Failure Stab Open Loop with Adaptation; 20) Canard Multiplier Effect Closed Loop with Adaptation; 21) Gen 2 NN Wts from Simulation; 22) Direct Adaptive Experience and Lessons Learned; and 23) Conclusions

Bosworth, John↗