Search NASA⌕ Search

SEARCH · Search NASA

Results for “model-free”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Adaptive Control of Distributed Energy Resources for Distribution Grid Voltage Stability

Volt-VAR and Volt-Watt functionality in photovoltaic (PV) smart inverters provide mechanisms to ensure system voltage magnitudes and power factors remain within acceptable limits. However, these control functions can become unstable, introducing oscillations in system voltages when not appropriately configured or maliciously altered during a cyberattack. In the event that Volt-VAR and Volt-Watt control functions in a portion of PV smart inverters in a distribution grid are unstable, the proposed adaptation scheme utilizes the remaining and stably-behaving PV smart inverters and other Distributed Energy Resources to mitigate the effect of the instability. The adaptation mechanism is entirely decentralized, model-free, communication-free, and requires virtually no external configuration. Here we provide a derivation of the adaptive control approach and validate the algorithm in experiments on the IEEE 37 and 8500 node test feeders.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

A Barrier-Certificated Reinforcement Learning Approach for Enhancing Power System Transient Stability

Increasing integration of renewable resources brings more flexibility and poses new challenges to modern power systems, leading to highly nonlinear and complex dynamics. Here, this paper aims to provide a general solution framework to traditional control problems, such as frequency control and voltage control, which attempt to maintain the stability of either synchronous generators-governed or inverter-governed systems when subjected to a disturbance and simultaneously guarantee operational constraints, providing a complete complement to existing works on control design. Building on reinforcement learning (RL) and control barrier functions, the framework includes two subsystems, i.e., a model-free controller and a barrier-certification system, which discover RL-based control actions and sequentially filter them using a barrier certificate to satisfy operational constraints. Calculating a barrier function is generally challenging for a complex power system. This is addressed by representing the barrier function using neural networks (NNs) and data-based approaches. An adaptive method is introduced to certify the neural barrier function that perseveres barrier conditions, which is more compatible with online implementation. The proposed framework synthesizes a stabilizing controller that satisfies predefined safety regions. The effectiveness of the proposed framework is demonstrated via several comparative case studies.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

Multiarea Distribution System State Estimation via Distributed Tensor Completion

Here, this paper proposes a model-free distribution system state estimation method based on tensor completion using canonical polyadic decomposition. In particular, we consider a setting where the network is divided into multiple areas. The measured physical quantities at buses located in the same area are processed by an area controller. A three-way tensor is constructed to collect these measured quantities. The measurements are analyzed locally to recover the full state information of the network. A distributed closed-form iterative algorithm based on the alternating direction method of multipliers is developed to obtain the low-rank factors of the whole network state tensor where information exchange happens only between neighboring areas. The convergence properties of the distributed algorithm and the sufficient conditions on the number of samples for each smaller network that guarantee the identifiability of the factors of the state tensor are presented. To demonstrate the efficacy of the proposed algorithm and to check the identifiability conditions, numerical simulations are carried out using the IEEE 123-bus system and a large-scale real utility feeder.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2024

Learning Constrained Parametric Differentiable Predictive Control Policies With Guarantees

We present differentiable predictive control (DPC), a method for offline learning of constrained neural control policies for nonlinear dynamical systems with performance guarantees. We show that the sensitivities of the parametric optimal control problem can be used to obtain direct policy gradients. Specifically, we employ automatic differentiation (AD) to efficiently compute the sensitivities of the model predictive control (MPC) objective function and constraints penalties. To guarantee safety upon deployment, we derive probabilistic guarantees on closed-loop stability and constraint satisfaction based on indicator functions and Hoeffding’s inequality. We empirically demonstrate that the proposed method can learn neural control policies for various parametric optimal control tasks. In particular, we show that the proposed DPC method can stabilize systems with unstable dynamics, track time-varying references, and satisfy nonlinear state and input constraints. Our DPC method has practical time savings compared to alternative approaches for fast and memory-efficient controller design. Specifically, DPC does not depend on a supervisory controller as opposed to approximate MPC based on imitation learning. We demonstrate that, without losing performance, DPC is scalable with greatly reduced demands on memory and computation compared to implicit and explicit MPC while being more sample efficient than model-free reinforcement learning (RL) algorithms.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Neural Networks-Based Inverter Control: Modeling and Adaptive Optimization for Smart Distribution Networks

The optimal voltage control of inverter-based resources, especially under the high penetration of solar photovoltaics, is critical to the stability of the distribution power system. However, the computational complexity as well as the coordinated operation performance of the voltage control optimization in the distribution power system limits the real-time applications. To mitigate this issue, a model-free based adaptive optimal control scheme for the smart inverter is proposed to maximize the active power generation, minimize the power loss, and maintain the bus voltages in smart distribution networks. An inverter-based optimization model for coordinated operation is first established, considering the uncertainties of renewable power generation. Subsequently, by collecting the data and control strategies, the neural networks (NNs) based algorithm is proposed to efficiently predict the best possible control strategy. The main objective of this scheme is to accurately predict candidate optimal solutions with near-negligible feasibility and optimization gaps, with the advantage of avoiding complicated iteration-based numerical algorithms. Thereafter, the co-simulation among OpenDSS, MATLAB, and Python is set up to fully take advantage of the three individual software. Experiments are conducted based on different control parameter characteristics and structures of NNs. Finally, the results reveal that an average mean squared error of 0.013 and 1 ms response time are achieved, which is lower than some state-of-the-art methods.

42 ENGINEERING↗
DOE OSTI2023

Automated Adversary-in-the-Loop Cyber-Physical Defense Planning

Security of cyber-physical systems (CPS) continues to pose new challenges due to the tight integration and operational complexity of the cyber and physical components. To address these challenges, this article presents a domain-aware, optimization-based approach to determine an effective defense strategy for CPS in an automated fashion—by emulating a strategic adversary in the loop that exploits system vulnerabilities, interconnection of the CPS, and the dynamics of the physical components. Our approach builds on an adversarial decision-making model based on a Markov Decision Process (MDP) that determines the optimal cyber (discrete) and physical (continuous) attack actions over a CPS attack graph. The defense planning problem is modeled as a non-zero-sum game between the adversary and defender. We use a model-free reinforcement learning method to solve the adversary’s problem as a function of the defense strategy. We then employ Bayesian optimization (BO) to find an approximate best-response for the defender to harden the network against the resulting adversary policy. This process is iterated multiple times to improve the strategy for both players. We demonstrate the effectiveness of our approach on a ransomware-inspired graph with a smart building system as the physical process. Numerical studies show that our method converges to a Nash equilibrium for various defender-specific costs of network hardening.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

OODG (OnlineOptDynaGrid) [SWR-24-07]

This is a repository for the code developed by NREL under the DynaGrid project. It is a Julia package for running a model-free real-time OPF algorithm to operate dynamic networked microgrids given network configuration results from the Maximal Load Delivery (MLD) problem in the PowerModelsONM.jl package. See Example.jl for a working example on how to use the package. Note: Julia package PowerModelsONM.jl is needed.

Poolla, Bala Kameshwar [National Renewable Energy ↗
DOE OSTI2022

graphenv: a Python library for reinforcement learning on graph search spaces

Many important and challenging problems in combinatorial optimization (CO) can be expressed as graph search problems, in which graph vertices represent full or partial solutions and edges represent decisions that connect them. Graph structure not only introduces strong relational inductive biases for learning (Battaglia et al., 2018) - in this context, by providing a way to explicitly model the value of transitioning (along edges) between one search state (vertex) and the next - but lends itself to problems both with and without clearly defined algebraic structure. For example, classic CO problems on graphs such as the Traveling Salesman Problem (TSP) can be expressed as either pure graph search or integer programs. Other problems, however, such as molecular optimization, do no have concise algebraic formulations and yet are readily implemented as a graph search (V. et al., 2022; Zhou et al., 2019). Such "model-free" problems constitute a large fraction of modern reinforcement learning (RL) research owing to the fact that it is often much easier to write a forward simulation that expresses all of the state transitions and rewards, than to write down the precise mathematical expression of the full optimization problem. In the case of molecular optimization, for example, one can use domain knowledge alongside existing software libraries to model the effect of adding a single bond or atom to an existing but incomplete molecule, and let the RL algorithm build a model of how good a given decision is by "experiencing" the simulated environment many times through. In contrast, a model-based mathematical formulation that fully expresses all the chemical and physical constraints is intractable. In recent years, RL has emerged as an effective paradigm for optimizing searches over graphs and led to state-of-the-art heuristics for games like Go and chess, as well as for classical CO problems such as the TSP. This combination of graph search and RL, while powerful, requires non-trivial software to execute, especially when combining advanced state representations such as Graph Neural Networks (GNN) with scalable RL algorithms.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Imaging Wavepackets in Real-Space & Disentangling Ultrafast Solvation Dynamics with High-Energy Ultrafast X-ray Scattering

The microscopic information of solute and solute-solvent motions can be measured using ultrafast diffuse x-ray scattering, and compared to molecular dynamics simulations, as demonstrated in studies of solvated molecular systems done in X-ray free-electron lasers. However, for the typical photon energies used in these experiments, the photon momentum transfer range was limited to lower Q values where the signals arising from different parts of the studied system overlap significantly. Recently, this limitation was greatly alleviated by extending the photon energy at the Linac Coherent Light-Source (LCLS), significantly increasing the accessible momentum transfer range. This improvement is transformative for ultrafast diffuse x-ray scattering measurements, enabling for the first time to disentangle the solute scattering difference signal, that persists to higher Q values, from the bulk solvent and solute-solvent cross-terms difference signals. In addition, it enhances the prediction ability of modeling and simulation methods such as the hybrid QM/MM approach. The extended Q range also opens the way to resolve in real-space details regarding coherent wavepackets motions beyond their center-of-mass positions. Here, we present the first results on high-energy (18keV) diffuse x-ray scattering in solution, demonstrating high fidelity time-resolved scattering and analysis of the photoexcited model photocatalysts PtPOP ([Pt2(POP)4]4-), and IrDimen ([Ir2(dimen)4]2+) in several solvents. These complexes provide ideal systems for demonstrating the ability of high-Q ultrafast scattering and QM/MM simulations to decompose ultrafast solvation dynamics into specific changes in the solute-solvent pair distribution function with a particular focus on how electronically excited states change the interaction between the solvent and photo-catalytically active metal sites. We introduce an approach to further utilize the high-energy capability and demonstrate a single-shot ultra-wide-angle X-ray Scattering modality using two perpendicular detectors, spanning a scattering angle range of more than 100 degrees, allowing to extend the accessible momentum transfer range up to Q~14 Å-1. We develop a model-free method to invert the x-ray scattering signals and enable the recovery of multiple pair-density motions that happen simultaneously, allowing us to directly measure in real-space nuclear wavepacket motions. . [1] Natan, Adi. "Real-Space Inversion and Super-Resolution of Ultrafast X-ray Scattering using Natural Scattering Kernels." arXiv preprint arXiv:2107.05576 (2021)

Natan, Adi↗
DOE OSTI2024

Achieving Cyber-Resilience for Power Systems using a Learning, Model-Assisted Blockchain Framework

The secure integration and management of distributed energy resources (DER) and power aggregators in the electric grid requires secure communications and a physics-aware Command and Control (C2) strategy. A Blockchain (BC)-based overlay network was developed to provide a security layer for the existing power grid network that mitigates risks in current and legacy network and C2 protocols. By integrating a Model-Assisted Machine Learning (MAML) framework with a Secure Blockchain Overlay Network (SBON) a defense-in-depth strategy was achieved. In our approach, the MAML framework leveraged a smart contract framework to gather network data and learn the dynamics of DER to develop detection strategies for attacks targeting sensors and actuators used by DER. The MAML framework learned dynamical systems models for individual DERs to detect sensor attacks. For DER we utilized a Digital Twin (DT) to accelerate the learning process for a model resistant to stealthy attacks. The project created DT for PV inverters and BESS. The DTs were coupled with a model-assisted, data-driven learning of DER behavior. Specifically, we evaluated architectures for model-based learning with model-free fine-tuning. Additionally, differential privacy techniques were used to obfuscate data, while still allowing the computation of attack detection results based on obfuscated data. The SBON developed leverages a private permissioned blockchain network orchestrated with the Hyperledger Fabric framework. To connect the cyber world, which orchestrates the blockchain fabric, and the physical world where the power network resides, we developed a system implementation to enable the secure interaction of the physical world and the abstracted blockchain.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Unraveling the Pyrolytic Behavior and Kinetics of Single Polymers and Plastic-Rich Municipal Solid Waste Using Thermal Analysis

Pyrolysis is a highly promising thermochemical recycling technology for converting heterogenous plastic waste into sustainable fuels in a single step. Therefore, understanding the pyrolysis mechanism is essential for enabling rational reactor design and enhancing efficient recycling techniques. In this study, the thermal degradation behaviors and corresponding kinetics of pure polymers (PE, PP, and PET) and plastic-rich MSW were examined using simultaneous thermogravimetric analysis (TGA) and differential scanning calorimetry (DSC). Experiments were carried out in the temperature range of 30-800°C with variable heating rates from 5°C/min to 20°C/min in an ultra-high purity Argon atmosphere. Our results indicated that the plastic pyrolysis was an endothermic process, with varying decomposition temperature ranges depending on their structure and composition. Various iso-conversional model-free methods (Friedman, Flynn-Wall-Ozawa, Starink, and Kissinger-Akahira-Sunose) were utilized to determine the apparent activation energy of the plastic degradation, which increased in the following order: PET (214 kJ/mol), PP (218 kJ/mol), PE (245 kJ/mol), and MSW (249 kJ/mol). Finally, Criado’s master plots were employed to identify the best-fitting reaction model and the pre-exponential factor was subsequently determined.

Bashir, Muhammad Aamir↗
DOE OSTI2021

Deep Reinforcement Learning for Autonomous Water Heater Control

Electric water heaters represent 14% of the electricity consumption in residential buildings. An average household in the United States (U.S.) spends about USD 400–600 (0.45 ¢/L–0.68 ¢/L) on water heating every year. In this context, water heaters are often considered as a valuable asset for Demand Response (DR) and building energy management system (BEMS) applications. To this end, this study proposes a model-free deep reinforcement learning (RL) approach that aims to minimize the electricity cost of a water heater under a time-of-use (TOU) electricity pricing policy by only using standard DR commands. In this approach, a set of RL agents, with different look ahead periods, were trained using the deep Q-networks (DQN) algorithm and their performance was tested on an unseen pair of price and hot water usage profiles. The testing results showed that the RL agents can help save electricity cost in the range of 19% to 35% compared to the baseline operation without causing any discomfort to end users. Additionally, the RL agents outperformed rule-based and model predictive control (MPC)-based controllers and achieved comparable performance to optimization-based control.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2022

SDSS-IV MaNGA: How the Stellar Populations of Passive Central Galaxies Depend on Stellar and Halo Mass

We analyze spatially resolved and co-added SDSS-IV MaNGA spectra with signal-to-noise ratio ~100 from 2200 passive central galaxies (z ~ 0.05) to understand how central galaxy assembly depends on stellar mass (M*) and halo mass (M h ). We control for systematic errors in M h by employing a new group catalog from Tinker and the widely used Yang et al. catalog. At fixed M*, the strengths of several stellar absorption features vary systematically with M h . Completely model-free, this is one of the first indications that the stellar populations of centrals with identical M* are affected by the properties of their host halos. To interpret these variations, we applied full spectral fitting with the code alf. At fixed M*, centrals in more massive halos are older, show lower [Fe/H], and have higher [Mg/Fe] with 3.5σ confidence. We conclude that halos not only dictate how much M* galaxies assemble but also modulate their chemical enrichment histories. Turning to our analysis at fixed M h , high-M* centrals are older, show lower [Fe/H], and have higher [Mg/Fe] for M h > 10 12 h –1 M⊙ with confidence >4σ. While massive passive galaxies are thought to form early and rapidly, our results are among the first to distinguish these trends at fixed M h . They suggest that high-M* centrals experienced unique early formation histories, either through enhanced collapse and gas fueling or because their halos were early forming and highly concentrated, a possible signal of galaxy assembly bias.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Lattice QCD and the Computational Frontier

The search for new physics requires a joint experimental and theoretical effort. Lattice QCD is already an essential tool for obtaining precise model-free theoretical predictions of the hadronic processes underlying many key experimental searches, such as those involving heavy flavor physics, the anomalous magnetic moment of the muon, nucleon-neutrino scattering, and rare, second-order electroweak processes. As experimental measurements become more precise over the next decade, lattice QCD will play an increasing role in providing the needed matching theoretical precision. Achieving the needed precision requires simulations with lattices with substantially increased resolution. As we push to finer lattice spacing we encounter an array of new challenges. They include algorithmic and software-engineering challenges, challenges in computer technology and design, and challenges in maintaining the necessary human resources. In this white paper we describe those challenges and discuss ways they are being dealt with. Overcoming them is key to supporting the community effort required to deliver the needed theoretical support for experiments in the coming decade.

Boyle, Peter↗
DOE OSTI2023

Primal-Dual Differentiable Programming for Distribution System Critical Load Restoration: Preprint

Swift and reliable critical load restoration (CLR) can help make a distribution system resilient towards extreme events. To optimally achieve that, alongside practical concerns such as limiting online computational burden, some studies leverage model-free reinforcement learning (RL) to train control policies. Despite the advantages provided by RL algorithms, these approaches suffer from two issues: 1) the lack of a proper mechanism for constraint enforcement, and 2) poor sample efficiency. Therefore, in this paper, a primal-dual differentiable programming (PDDP) method is developed for guiding the training leading to a constraint-satisfying policy. Additionally, the model-based nature of the proposed method aims at improving sample efficiency. The experiment on a CLR problem demonstrates that PDDP can effectively train a control policy that both achieves desirable performance and satisfies required constraints.

differentiable programming↗
DOE OSTI2023

A Novel Multi-Agent Deep Reinforcement Learning-enabled Distributed Power Allocation Scheme for mmWave Cellular Networks

We consider the power allocation problem over shared spectrum for millimeter-Wave (mmWave) cellular downlink. Existing approaches usually find sub-optimal solutions by solving a non-convex optimization which leads to scalability issues due to centralized control. Therefore, distributed and adaptive approaches are desirable. Recently, model-free Deep Reinforcement Learning (DRL) has achieved success in such wireless resource management tasks. By modeling the radio environment as a Markov Decision Process (MDP) with the base stations (BSs) being the agents, power allocation can be automated at the agent level with comparable throughput performance to conventional centralized schemes. The multi-agent setting presents new challenges as the radio environment is impacted by the joint actions of the agents and is no longer stationary from any individual agent’s perspective. Existing literature bypasses this non-stationarity violation by ignoring it which may cause performance degradation. To tackle this issue, we propose a distributed continuous power allocation scheme based on a modified version of multi-agent Deep Deterministic Policy Gradient (MADDPG) that is tailored for the distributed multiple-agent setting. The proposed scheme employs a centralized-training distributed-execution framework where Q-functions are trained over subsets of BSs while each BS determines its transmit power based only on its own local observation. It admits constant per-BS communication and computation complexity and is thus scalable to large networks. Numerical evaluation shows that the proposed scheme adapts well to a wide range of interference conditions and can achieve comparable or better performance than several state-of-the-art non-learning approaches.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2024

Reinforcement Learning to Enhance Optimal Operation of Resilient Community Energy Systems

This paper presents a novel model-free multi-agent Reinforcement Learning (RL) control method to enhance the resilience of community energy systems in island mode, which coordinates multiple objectives without the necessity of identifying system models that require expert knowledge. Specifically, a community-level coordinator agent is designed to allocate renewable energy resources among different buildings, and multiple building-level agents are developed to optimize load schedules based on limited energy resources and requirements of building loads and occupants’ comfort. In a two-day evaluation, our RL approach demonstrated a similar performance against MPC without requiring system models and formulation of optimization problems as required in MPC.

ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATION↗