Search NASA⌕ Search

SEARCH · Search NASA

Results for “practical reinforcement learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

AI-Enabled Discovery and Physics-Based Optimization of Energy Efficient Processing Strategies for Advanced Turbine Alloys (Final Technical Report)

In this project, the multi-organizational team of academic and industrial researchers from the University of Kentucky an aerospace and energy generation OEM partner has leveraged novel Digital Process Twin (DPT) models of process/structure interactions (i.e., process-induced surface integrity) to advance a paradigm of fully integrated computational materials engineering (ICME). Using efficient process models as the core of a digital process simulator for a reinforcement learning algorithm, the team has integrated industrial data and metrics of structure/performance/energy relationships and manufacturing-related energy metrics to optimize dynamic processing parameters for significantly improved life-cycle energy efficiency of advanced γ-TiAl low-pressure turbine (LPT) alloys, as indicated by a set of design relevant parameters (e.g., residual stresses and scrap rate). The key objective and anticipated outcome of the project was at least a 10% reduction in life-cycle embodied energy for a recently developed, γ-TiAl low-pressure turbine (LPT) alloy and nickel-based superalloy Inconel 718, through the adoption of the proposed AI-enabled process optimization approach. The final project outcomes significantly exceeded this original target, realizing manufacturing-related energy efficiency improvements of more than 130% for TiAl and up to 80% for Inconel 718. Rather than following the prevailing and highly inefficient empirical paradigm, the proposed study demonstrated the feasibility of adopting a digital, physics-based process design and optimization paradigm. The recurring need for manual intervention, rework, reinspection causes significant production bottlenecks and unnecessary expense associated with delivering the requisite component quality. The OEM partner, and turbine industry in general, expect to reap significant cost and resource savings if an AI-optimized set of parameters can be applied to specific machining operations. The technical scope of the proposed project involved the paving of a realistic path towards model-based and AI-enabled Integrated Computational Materials Engineering (ICME), and away from inefficient empirical process optimization and legacy manufacturing practices, which are no longer able to efficiently process novel high-performance turbine alloy materials. The project team will address the fundamental knowledge gap that currently exists within the ICME paradigm with respect to the process/structure/performance/energy impacts of finishing processes. While significant resources have been devoted to the ‘early stages’ of manufacturing, such as alloy design, primary and secondary processing, finishing processes have not been adequately integrated within ICME. To provide an actionable path towards model-based finishing process design (e.g., machining, burnishing, grinding, polishing), we will employ a novel AI-enabled process optimization paradigm, based on a computationally efficient, physics-based process simulator. Through limited experimental work to calibrate and validate our process simulator model via an advanced in-situ characterization technique and process optimization via reinforcement learning, the project will seek to demonstrate a viable alternative to the inefficient ‘legacy’ processing strategies, empirical testing and broad scope machining learning approaches, all of which fail to adequately consider complex process physics. The project team has identified an intermetallic γ-TiAl LPT alloy, which is currently being used as part of the OEM partner’s advanced gas turbine designs. This particular alloy poses significant manufacturing challenges during finishing operations, which limit the degree to which the current turbine design can be manufactured in an energy- and cost-efficient manner. Empirical testing and numerical modeling efforts to optimize processing parameters for γ-TiAl have not been able to resolve these manufacturing challenges, so the proposed physics-based AI-enabled optimization technology would offer a truly novel and transformative capability. The multi-organizational team of academic and industry experts from the UKY and the OEM partner will work together closely to demonstrate the analytical and experimental critical function and characteristic proof of concept of this novel approach.

20 FOSSIL-FUELED POWER PLANTS↗
DOE OSTI2022

A resilient network recovery framework against cascading failures with deep graph learning

Because of the increasing importance and dependencies of infrastructure networks and the potential for massive cascading failures in real-world network systems, maintenance optimization to effectively reduce system performance loss caused by diverse disruptions is of significant interest among researchers and practitioners. In this work, a new recovery framework was developed to rapidly identify important system components for maintenance to improve network resilience against cascading failures. Here this work provides distinct advantages to determine an optimal maintenance priority by combining real-time network structure importance with other maintenance prioritization based on customer preference. This approach adopts structural graph embedding and deep reinforcement learning to extract real-time network topology information (such as minimum vertex cover) to update the maintenance priority during the recovery process. Based on the case studies on synthetic networks and a US airport network, the proposed recovery framework with real-time network topology awareness shows better performance than other maintenance prioritization strategies regarding resilience enhancement. This work improves the understanding of how the changing network structure influences maintenance effects. It also provides insights of the practical usefulness of advanced deep learning on helping optimal maintenance prioritization to effectively reduce the intensity and extent of cascading failures.

42 ENGINEERING↗
DOE OSTI2022

A reinforcement learning approach to long-horizon operations, health, and maintenance supervisory control of advanced energy systems

In this work, we develop a Reinforcement Learning (RL) approach to the supervisory control problem for advanced energy systems, such as novel nuclear reactors and other demand-driven, mission-critical, and component-health-sensitive energy plants. The inclusive problem landscape considered captures the stochastic confluence of plant performance, component health evolution, power demand from the grid, diverse maintenance actions, and operator-defined goals and constraints, all considered over meaningfully long-enough reasoning horizons. Key aspects of the proposed approach are a receding horizon control-inspired technique dictating time- or event-triggered supervisory policy (re-)constructions, as well as additional capability-enabling contributions such as timescale compression, to handle long reasoning horizons and uncertainty in parts of the problem, and practical yet demonstrably-effective handling of hybrid action spaces with continuous and discrete decision variables. The resulting algorithm consists of a simulation-based RL agent constructing stochastic supervisory control policies over nontrivial action spaces and for long horizons, applying the learned policy to the system for a much shorter interval, and perpetually repeating, to construct the next long-horizon policy. That next policy will only be applied, again, for a short interval, yet originally far-in-time events move progressively closer, their associated uncertainty decreases, and new events and aspects enter the reasoning horizon. The proposed methodology bridges fundamental receding horizon concepts with the unequivocally stronger and more scalable reasoning of contemporary RL. Numerical examples using Soft Actor–Critic Deep RL illustrate the operation and efficacy of the proposed technique for a power plant tasked with health-aware load following missions in a dynamic electricity market landscape.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Adaptive nonequilibrium design of actin-based metamaterials: Fundamental and practical limits of control

The adaptive and surprising emergent properties of biological materials self-assembled in far-from-equilibrium environments serve as an inspiration for efforts to design nanomaterials. In particular, controlling the conditions of self-assembly can modulate material properties, but there is no systematic understanding of either how to parameterize external control or how controllable a given material can be. Here, we demonstrate that branched actin networks can be encoded with metamaterial properties by dynamically controlling the applied force under which they grow and that the protocols can be selected using multi-task reinforcement learning. These actin networks have tunable responses over a large dynamic range depending on the chosen external protocol, providing a pathway to encoding “memory” within these structures. Interestingly, we obtain a bound that relates the dissipation rate and the rate of “encoding” that gives insight into the constraints on control—both physical and information theoretical. Taken together, these results emphasize the utility and necessity of nonequilibrium control for designing self-assembled nanostructures.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

FY20 Proxy App Suite Release: Report for ECP Proxy App Project Milestone ADCD-504-10

Version 4.0 of the ECP Proxy App Suite is practically unchanged from the previous release. The current set of proxies has proven useful for many aspects of benchmarking and co-design and we see little reason to alter the suite. Although there have been few changes to the ECP suite, the team has been hard at work in other areas. In the area of Machine Learning (ML) we have now created a separate proxy suite dedicated to this scientific applications of ML. The suite includes: miniGAN (Generative Adversarial Networks), miniRL (Reinforcement Learning), CRADL (inline inference), Cosmoflow-Benchmark (Convolutional Neural Network), and MLPerf-DeepCam (Climate Segmentation Benchmark). Section 3 contains more information about these proxies as well as the principles that are guiding the development of the suite. We have surveyed available proxies for several application domains including Computational Fluid Dynamics, Quantum Chemistry, Quantum Computing Simulation, Molecular Dynamics, Monte Carlo Transport, and Density Functional Theory to identify gaps in proxy coverage. Several new proxy apps are either already available or will be released soon to fill these gaps. Section 4 provides full details. Finally, section 5 reports on our continued collaboration with the ECP Continuous Integration (CI) effort to use proxy apps to help identify problems and roadblocks to cross-lab CI. We have also assisted the El Capitan Center of Excellence (COE) to stand up a CI system that will be used to test releases of HPE and AMD software stacks. We hope that the COE effort can serve as a model for ECP by showing how the Proxy App Team can work with AD and ST teams identify critical features, kernels, patterns, etc. and incorporate them into a CI system that will help ensure that Frontier and Aurora will provide those needed capabilities.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Real-time quantum error correction beyond break-even

The ambition of harnessing the quantum for computation is at odds with the fundamental phenomenon of decoherence. The purpose of quantum error correction (QEC) is to counteract the natural tendency of a complex system to decohere. This cooperative process, which requires participation of multiple quantum and classical components, creates a special type of dissipation that removes the entropy caused by the errors faster than the rate at which these errors corrupt the stored quantum information. Previous experimental attempts to engineer such a process faced an excessive generation of errors that overwhelmed the error-correcting capability of the process itself. Whether it is practically possible to utilize QEC for extending quantum coherence thus remains an open question. We answer it by demonstrating a fully stabilized and error-corrected logical qubit whose quantum coherence is significantly longer than that of all the imperfect quantum components involved in the QEC process, beating the best of them with a coherence gain of G=2.27±0.07. Here we achieve this performance by combining innovations in several domains including the fabrication of superconducting quantum circuits and model-free reinforcement learning.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2020

RLScheduler: An Automated HPC Batch Job Scheduler Using Reinforcement Learning

Today’s high-performance computing (HPC) platforms are still dominated by batch jobs. Accordingly, effective batch job scheduling is crucial to obtain high system efficiency. Existing HPC batch job schedulers typically leverage heuristic priority functions to prioritize and schedule jobs. But, once configured and deployed by the experts, such priority functions can hardly adapt to the changes of job loads, optimization goals, or system settings, potentially leading to degraded system efficiency when changes occur. To address this fundamental issue, we present RLScheduler, an automated HPC batch job scheduler built on reinforcement learning. RLScheduler relies on minimal manual interventions or expert knowledge, but can learn high-quality scheduling policies via its own continuous ‘trial and error’. We introduce a new kernel-based neural network structure and trajectory filtering mechanism in RLScheduler to improve and stabilize the learning process. Through extensive evaluations, we confirm that RLScheduler can learn high-quality scheduling policies towards various workloads and various optimization goals with relatively low computation cost. Moreover, we show that the learned models perform stably even when applied to unseen workloads, making them practical for production use.

Zhang, Di↗
DOE OSTI2025

A Two-Stage Quantum Reinforcement Learning Method for Multi-Objective Transmission Switching

Multi-objective transmission switching (MO-TS) problems involve the strategic reconfiguration of network topology to simultaneously optimize multiple objectives. As the system scale increases, finding feasible solutions becomes increasingly challenging due to the problem's nonlinearity and high computational complexity. To address these challenges, this paper proposes a two-stage quantum reinforcement learning method that leverages potential quantum advantages for MO-TS. In the first stage, candidate switching lines are identified using a graph-theoretical approach to reduce the problem's dimensionality. The second stage introduces a quantum-classical reinforcement learning framework, where a learnable measurement-based CNN-ResVQC architecture is developed to effectively reduce the input dimension for quantum processing, mitigate vanishing gradients, and enhance trainability while improving the quantum circuit's flexibility in modeling complex decision policies for MO-TS. Numerical studies on IEEE 14-bus, 57-bus, and 118-bus systems demonstrate that the proposed algorithm achieves superior training stability and faster convergence with approximately 1% of the network parameters required by classical algorithms, highlighting its effectiveness, efficiency, and scalability. Furthermore, the practicality is validated through its stable convergence under three common quantum noise channels.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2025

Reinforcement Learning-Based Oscillation Dampening: Scaling Up Single-Agent Reinforcement Learning Algorithms to a 100-Autonomous-Vehicle Highway Field Operational Test

In this article, we explore the technical details of the reinforcement learning (RL) algorithms that were deployed in the largest field test of automated vehicles designed to smooth traffic flow in history as of 2023, uncovering the challenges and breakthroughs that come with developing RL controllers for automated vehicles. We delve into the fundamental concepts behind RL algorithms and their application in the context of self-driving cars, discussing the developmental process from simulation to deployment in detail, from designing simulators to reward function shaping. We present the results in both simulation and deployment, discussing the flow-smoothing benefits of the RL controller. From understanding the basics of Markov decision processes to exploring advanced techniques such as deep RL, our article offers a comprehensive overview and deep dive of the theoretical foundations and practical implementations driving this rapidly evolving field. We also showcase real-world case studies and alternative research projects that highlight the impact of RL controllers in revolutionizing autonomous driving. From tackling complex urban environments to dealing with unpredictable traffic scenarios, these intelligent controllers are pushing the boundaries of what automated vehicles can achieve. Furthermore, we examine the safety considerations and hardware-focused technical details surrounding deployment of RL controllers into automated vehicles. As these algorithms learn and evolve through interactions with the environment, ensuring their behavior aligns with safety standards becomes crucial. Here, we explore the methodologies and frameworks being developed to address these challenges, emphasizing the importance of building reliable control systems for automated vehicles.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
NASA NTRS1972

Degradation of learned skills: Effectiveness of practice methods on visual approach and landing skill retention

Flight control and procedural task skill degradation, and the effectiveness of retraining methods were evaluated for a simulated space vehicle approach and landing under instrument and visual flight conditions. Fifteen experienced pilots were trained and then tested after 4 months either without the benefits of practice or with static rehearsal, dynamic rehearsal or with dynamic warmup practice. Performance on both the flight control and procedure tasks degraded significantly after 4 months. The rehearsal methods effectively countered procedure task skill degradation, while dynamic rehearsal or a combination of static rehearsal and dynamic warmup practice was required for the flight control tasks. The quality of the retraining methods appeared to be primarily dependent on the efficiency of visual cue reinforcement.

Sitterley, T. E.↗
DOE OSTI2023

Quick Talk: Resilient Communities, Maryland

Groundswell, Inc., in partnership with the Maryland Energy Administration (MEA), Ayika Solutions, the National Renewable Energy Lab (NREL), and the Baltimore Office of Sustainability, is advancing a vision for resilient, community-centered energy systems in Maryland. This effort aligns with Groundswell's commitment to a clean energy economy that equitably benefits all communities, closing gaps in representation, wages, and access to wealth generation, savings, and local jobs within a diverse energy supply chain. Project Vision and Objectives The project emphasizes "energy resilience," defined as a community’s capacity to anticipate, adapt to, and swiftly recover from power disruptions through inclusive planning and technical solutions. The approach centers on Resilience Hubs—designated community facilities equipped to support residents before, during, and after power-related disruptions. These hubs offer emergency resources, coordinate communications, and deliver essential services while reducing carbon emissions, ultimately enhancing residents' quality of life. Additionally, Resilience Hubs foster social connections, support disaster preparedness, and provide spaces for community programming and everyday use. The goals of Resilient Communities Maryland include: Developing a replicable framework for community energy resilience, applicable beyond Maryland. Defining community-led indicators of social impact and resilience. Establishing equitable decision-making processes in project design and implementation. Informing MEA of practical, community-based resilience strategies to shape future state initiatives. Approach Groundswell and its partners prioritize collaborative engagement with local stakeholders, ensuring that solutions are responsive to each community's unique needs. By integrating feedback from residents, local organizations, and government entities, the team has developed a holistic model for resilience that balances immediate benefits, such as improved emergency response, with long-term gains in social equity and environmental sustainability. Key Outcomes This initiative has produced valuable resources and insights, including: A Community Energy Resilience Framework: Designed to guide other communities in establishing energy resilience strategies tailored to their specific needs. Case Studies: Documenting best practices and lessons learned from pilot projects to serve as models for future resilience efforts. Community-Driven Impact Metrics: Highlighting the positive social impacts that resilience initiatives can bring, such as improved social connectedness and local decision-making power. Recommendations for MEA: Offering evidence-based insights to inform policies and funding opportunities that support resilient energy infrastructure in vulnerable communities. Next Steps Groundswell's final report will detail findings, including the potential for scaling resilience hubs across Maryland and other regions, with a focus on equitable access and community-driven outcomes. This project underscores the importance of aligning energy resilience with social equity, reinforcing community agency, and building systems that withstand and recover from future challenges effectively and sustainably.

14 SOLAR ENERGY↗
NASA NTRS1993

A proposal of an architecture for the coordination level of intelligent machines

The issue of obtaining a practical, structured, and detailed description of an architecture for the Coordination Level of Center for Intelligent Robotic Systems for Sapce Exploration (CIRSSE) Testbed Intelligent Controller is addressed. Previous theoretical and implementation works were the departure point for the discussion. The document is organized as follows: after this introductory section, section 2 summarizes the overall view of the Intelligent Machine (IM) as a control system, proposing a performance measure on which to base its design. Section 3 addresses with some detail implementation issues. An hierarchic petri-net with feedback-based learning capabilities is proposed. Finally, section 4 is an attempt to address the feedback problem. Feedback is used for two functions: error recovery and reinforcement learning of the correct translations for the petri-net transitions.

Beard, Randall↗
NASA NTRSDate not supplied

Runway Configuration Management with Offline Reinforcement Learning

Runway configuration management (RCM) is a challenging task, and it affects the efficiency of the National Airspace System (NAS) and airport surface operations significantly. Each airport, depending on the geometry, capacity, local climate patterns, etc. has multiple configurations for the runway usage for arriving and departing flights. Many factors such as the incoming/outgoing traffic load, wind direction and speed, convective weather, cloud ceiling and other environmental factors might affect the choice of a runway configuration at any point in time. However, other factors such as safety measures and regulations, noise abatement, capacity of each configuration, and preference of the air traffic controllers (ATCs) can also play a significant role in selecting the configuration. A sub-optimal selection of the runway configuration, or delay in making configuration changes might result in significant increase in taxi times for aircraft on the surface of the airport, fuel and energy use of the aircraft, and maintenance costs. It can also lead to safety concerns, such as an aircraft performing one or more go-arounds before being able to land. All these factors make RCM an extremely important and challenging decision-making process for the ATCs. The current state of practice sets the runway configuration by the ATCs based on relevant information available at the time including weather, traffic, noise abatement, safety bounds, etc. This makes the decision-making process subjective based on the accuracy of the available information and the bias in human decision making. Unfortunately, this approach yields poor results (e.g., significant delays) if the predicted outcomes are uncertain and their relative impact is not well understood. This is especially evident when the uncertainty increases the size of possible predicted outcomes (combinatorial explosion in possible scenarios) that cannot be handled by human reasoning. On the other hand, an automated approach based on machine intelligence can make use of historical data and search through all (or significant amount of) possible scenarios under uncertainty and make well-informed decisions.

Milad Memarzadeh↗
DOE OSTI2024

"Reinforcement Learning Based Oscillation Dampening: Scaling up Single-Agent RL algorithms to a 100 AV highway field operational test"

In this article, we explore the technical details of the reinforcement learning (RL) algorithms that were deployed in the largest field test of automated vehicles designed to smooth traffic flow in history as of 2023, uncovering the challenges and breakthroughs that come with developing RL controllers for automated vehicles. We delve into the fundamental concepts behind RL algorithms and their application in the context of self-driving cars, discussing the developmental process from simulation to deployment in detail, from designing simulators to reward function shaping. We present the results in both simulation and deployment, discussing the flow-smoothing benefits of the RL controller. From understanding the basics of Markov decision processes to exploring advanced techniques such as deep RL, our article offers a comprehensive overview and deep dive of the theoretical foundations and practical implementations driving this rapidly evolving field. We also showcase real-world case studies and alternative research projects that highlight the impact of RL controllers in revolutionizing autonomous driving. From tackling complex urban environments to dealing with unpredictable traffic scenarios, these intelligent controllers are pushing the boundaries of what automated vehicles can achieve. Furthermore, we examine the safety considerations and hardware-focused technical details surrounding deployment of RL controllers into automated vehicles. As these algorithms learn and evolve through interactions with the environment, ensuring their behavior aligns with safety standards becomes crucial. We explore the methodologies and frameworks being developed to address these challenges, emphasizing the importance of building reliable control systems for automated vehicles.

Jang, Kathy↗
NASA NTRS2019

Trends in Human Spaceflight: Failure Tolerance, High Reliability and Correlated Failure History

In a half century of human spaceflight, NASA has continuously refined agency safety and reliability requirements in response to mission demands, critical failures, and technology development. Early spacecraft, including Mercury, Gemini and Apollo vehicles, were highly reliant on dissimilar redundancy and demonstrated test margins. Later programs, such as the reusable Space Transportation System (STS) and International Space Station (ISS), introduced probabilistic studies and isolated two-failure tolerance to improve robustness at the expense of added complexity. More recently, the Orion Multi-Program Crew Vehicle (MPCV) program adopted universal single-failure tolerance with two categorical exceptions; Zero-Failure Tolerant (0FT) and Design for Minimum Risk (DFMR) hardware. Failure tolerance variances are defined and managed in accordance with agency human-rating requirements, and require concurrence from program Technical Authorities (TA) as well as the MPCV Safety and Mission Assurance Safety and Engineering Review Panel (MSERP). To understand and reaffirm standards applied to Apollo, Space Shuttle and Orion vehicles, Orion and Deep Space Gateway Safety and Mission Assurance (S&MA) representatives conducted accelerated research to compare unique safety and reliability criteria against ground and flight anomalies, based on information contained in post-mission reports and the Problem Reporting and Corrective Action (PRACA) database. In some cases, high-profile failures and narrow escapes have reinforced decisions to maintain or adapt safety requirements. In others, empirical trends have highlighted the need for vigilance and innovative safety guidelines. Given the inability to achieve absolute compliance with evolving safety and reliability requirements, the team conducted a targeted review of DFMR and 0FT propulsion elements within the framework of changing system design, inspection, materials and process developments to formulate conclusions on technological maturity, failure density, and net changes in safety risk. Based on the aggregate performance of high-reliability and failure-tolerant systems, the authors have attempted to establish best practices and guidelines to inform future program decisions. On a somewhat cautionary note, this study is not intended to direct a universal set of requirements for future missions based on prior lessons learned. Spacecraft safety is a multi-variable problem, and attempts to mitigate past failures will not guarantee future success. However, this assessment offers a retrospective review of policy changes, implementation and effectiveness. In the future, NASA, European Space Agency (ESA) and industry partners may benefit from a more robust correlation between requirements and performance, as space-faring nations work toward more challenging, complex and long-duration commercial and deep-space ventures.

Green, Carrie↗
DOE OSTI2021

Learning-based framework for sensor fault-tolerant building HVAC control with model-assisted learning

As people spend up to 87% of their time indoors, intelligent Heating, Ventilation, and Air Conditioning (HVAC) systems in buildings are essential for maintaining occupant comfort and reducing energy consumption. These HVAC systems in smart buildings rely 'on real-time sensor readings, which in practice often suffer from various faults and could also be vulnerable to malicious attacks. Such faulty sensor inputs may lead to the violation of indoor environment requirements (e.g., temperature, humidity, etc.) and the increase of energy consumption. While many model-based approaches have been proposed in the literature for building HVAC control, it is costly to develop accurate physical models for ensuring their performance and even more challenging to address the impact of sensor faults. In this work, we present a novel learning-based framework for sensor fault-tolerant HVAC control, which includes three deep learning based components for 1) generating temperature proposals with the consideration of possible sensor faults, 2) selecting one of the proposals based on the assessment of their accuracy, and 3) applying reinforcement learning with the selected temperature proposal. Moreover, to address the challenge of training data insufficiency in building-related tasks, we propose a model-assisted learning method leveraging an abstract model of building physical dynamics. Through extensive experiments, we demonstrate that the proposed fault-tolerant HVAC control framework can significantly reduce building temperature violations under a variety of sensor fault patterns while maintaining energy efficiency.

Xu, Shichao↗
NASA NTRS2011

Cognitive Behavioral Training and Education for Spaceflight Operations

Cognitive behavioral-training (CBT) is an evidence-based practice commonly used to help treat insomnia, and is part of NASA's countermeasure regimen for Fatigue Management. CBT addresses the life style and habits of individuals that are maladaptive to managing stress and fatigue. This includes addressing learned behavioral responses that may cause stress and lead to an increased sense of fatigue. While the initial cause of onset of fatigue in the individual may be no longer present, the perception and engrained anticipation of fatigue persist and cause an exaggerated state of tension. CBT combined with relaxation training allows the individual to unlearn the maladaptive beliefs and behaviors and replace them with routines and techniques that allow cognitive restructuring and resultant relief from stress. CBT allows for elimination in individuals of unwanted ruminating thoughts and anticipatory anxiety by, for example, training the individuals to practice stressful situations in a relaxed state. As a result of CBT, relaxation can be accomplished in many ways, such as progressive muscle relaxation, meditation and guided imagery. CBT is not therapy, but rather the synthesis of behavioral countermeasures. CBT utilizes progressive relaxation as a means of reinforcing educational and cognitive countermeasures. These countermeasures include: masking, elimination of distracting thoughts, anxiety control, split attention, cognitive restructuring and other advanced psychological techniques.

Moonmaw, Ronald↗
DOE OSTI2025

Designing reinforcement learning algorithms for building HVAC control: From experimental observation to simulation comparisons

Advanced supervisory-level control with reinforcement learning (RL) is regarded as a promising solution for HVAC systems to minimize energy consumption while maintaining thermal comfort and indoor air quality. However, most RL applications were conducted in the simulation environment rather than real-world HVAC systems. This paper developed a value-based RL controller termed Deep Q-Network (DQN) for a typical central HVAC system and evaluated its performance in a building test facility. By comparing DQN with a rule-based controller, the study not only demonstrated the cases where DQN could properly maintain indoor comfort but also discussed possible reasons why DQN failed in some other situations. Recognizing the limitations of value-based RL algorithms from the experimental tests, a simulation study was conducted to compare DQN with an alternative RL approach, an actor–critic algorithm termed Deep Deterministic Policy Gradient (DDPG). In scenarios with a relatively large action space, DDPG outperformed DQN by requiring fewer computational resources and achieving better thermal comfort, lower energy consumption, and more stable control actions. The findings suggest that the ability of DDPG to handle continuous control variables more effectively allows for faster convergence in training and more precise control in practice, which enhances the overall efficiency and reliability of the HVAC system.

Guo, Fangzhou↗