Resilient Operation of Distribution Grids Using Deep Reinforcement Learning
Not provided.
SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Not provided.
General Motors, LLC and energy partner Shell Hydrogen, LLC, deployed a system of hydrogen fuel cell electric vehicles integrated with a hydrogen fueling station infrastructure to operate under real world conditions as part of the U.S. Department of Energy's Controlled Hydrogen Fleet and Infrastructure Validation and Demonstration Project. This technical report documents the performance and describes the learnings from progressive generations of vehicle fuel cell system technology and multiple approaches to hydrogen generation and delivery for vehicle fueling.
The adaptive and surprising emergent properties of biological materials self-assembled in far-from-equilibrium environments serve as an inspiration for efforts to design nanomaterials. In particular, controlling the conditions of self-assembly can modulate material properties, but there is no systematic understanding of either how to parameterize external control or how controllable a given material can be. Here, we demonstrate that branched actin networks can be encoded with metamaterial properties by dynamically controlling the applied force under which they grow and that the protocols can be selected using multi-task reinforcement learning. These actin networks have tunable responses over a large dynamic range depending on the chosen external protocol, providing a pathway to encoding “memory” within these structures. Interestingly, we obtain a bound that relates the dissipation rate and the rate of “encoding” that gives insight into the constraints on control—both physical and information theoretical. Taken together, these results emphasize the utility and necessity of nonequilibrium control for designing self-assembled nanostructures.
The Robbins-Monro stochastic approximation algorithm is a foundation of many algorithmic frameworks for reinforcement learning and often an efficient approach to solving (or approximating the solution to) complex optimal control problems. In many cases, however, practitioners are unable to apply these techniques because of an inherent high variance. This paper aims to provide a general foundation for 'quasi-stochastic approximation,' in which all of the processes under consideration are deterministic, much like quasi-Monte Carlo for variance reduction in simulation. The variance reduction can be substantial, subject to tuning of pertinent parameters in the algorithm. This paper introduces a new coupling argument to establish the optimal rate of convergence provided the gain is sufficiently large. These results are established for linear models and tested also in nonideal settings. A major application of these general results is a new class of reinforcement learning algorithms for deterministic state space models. In this setting, the main contribution is a class of algorithms for approximating the value function for a given policy, using a different policy designed to introduce exploration.
In the development of a digital twin (DT) for hydropower turbines, dynamic modeling of the system (e.g., penstock, turbine, speed control) is crucial, along with all the necessary data interface, virtualization, and dashboard designs. Since the DT must mimic the actual dynamics of the hydropower turbine accurately, adaptive learning is required to train these dynamic models online so that the models in the DT can effectively follow the representation of the actual hydropower turbine dynamics accurately and reliably. This study presents an adaptive learning method for obtaining the hydropower turbine models for DT development of hydropower systems using the recursive least squares algorithm. To simplify the formulation, the hydropower turbine under consideration was assumed to operate near a fixed operating point, where the system dynamics can be well represented by a set of linear differential equations with constant parameters. In this context, the well-known six-coefficient model for the Francis turbine was formulated as the starting point to obtain input and output models for the turbine. Then, an adaptive learning mechanism was developed to learn model parameters using real-time data from a hydropower turbine testing system. This led to semi-physical modeling, in which first principles and data-driven modeling are integrated to produce dynamic models for DT development. Applications to a pilot system at the Norwegian University of Science and Technology (NTNU) were made, and the models learned adaptively using the data collected from the university’s pilot system. Desired modeling and validation results were obtained.
Smart building technologies are a new suite of resources that improve building energy efficiency and resilience, reduce carbon emissions, and provide load flexibility to the grid. However, in both college curricula and building professionals’ continuing education, there is a lack of systematic instruction on smart building technologies–topics that include smart building concepts, key components, smart building controls, “Internet of Things” (IoT) devices, and how to integrate multiple energy systems including distributed energy resources (DER). This major gap in smart building education prevents stakeholders from understanding and adopting smart building technologies in building design and operations. Slipstream leads a DOE-funded project developing a semester-long smart building curriculum for college students and adapting the contents into 16 training videos for building professionals and the general public. The education and training cover the drivers and benefits of smart building technologies, key building energy systems, the latest sensor technologies and IoT devices, and focus on topics related to smart building controls (i.e., energy management information systems, smart building control platforms, cybersecurity, grid-interactive-efficient buildings (GEBs), smart building control methods, and occupant-centric control. This paper describes the project approach, provides outlines of the training materials, and identifies lessons learned in creating the content. We also suggest ways to scale the instruction of smart building concepts to empower the workforce to accelerate the adoption of smart building technologies in the real world.
Navigating through the world of Artificial Intelligence (AI) in nuclear reactors and their Instrumentation and Control (I&C) systems demands a careful, deliberate journey. AI’s capability to manage massive datasets and streamline control systems has indeed carved out a significant role in various sectors, including nuclear energy. However, while AI, and particularly Large Language Models (LLMs), bring a lot to the table in terms of operational efficiency and anomaly detection, they also expose the sector to a new breed of cybersecurity threats, like Inference Attacks, Adversarial Attacks, and Trojan Attacks. This guide is designed to be a straightforward manual, diving deep into the intertwining worlds of AI and cybersecurity within nuclear reactors, and tailoring insights for three crucial audiences: I&C Vendors/Developers, Nuclear Regulators, and Nuclear Reactor Operators and Cyber Defense Teams. (1) Section 2, directed at I&C Vendors/Developers, will provide a clear and focused look at several cybersecurity attacks, offering practical recommendations and detailed scenarios related to AI cybersecurity. This section isn’t just about identifying problems but also about giving solid, usable solutions. (2) Section 3, meant for Nuclear Regulators, gets straight to the point about regulations, policy suggestions, and guidelines that are needed to lay down a robust, secure, and ethical foundation for the application of AI in nuclear operations. The focus is on making sure that everything adheres to international standards and laws while being practicable and clear-cut. (3) Section 4, aimed at Nuclear Reactor Operators and Cyber Defense Teams, offers an exhaustive exploration and technical reports, with clear recommendations and scenario analyses vital to protect operational environments and guarantee the secure application of AI in nuclear reactor operations. The goal is simple: as we step into an era where AI becomes a fundamental element of our technological and energy infrastructures, this guide is here to act as a clear, direct handbook, ensuring that AI is implemented within the nuclear sector in a manner that is secure, responsible, and practical. It’s about striking a balance – optimizing the undeniable benefits offered by AI while securing and shielding against potential cyber threats as we move through this new and complex landscape.
Blackbox modelling for SC analysis is a possible solution. Accuracy can be acceptable even without having vendor control diagrams. Blackbox modelling is not as user friendly as a generic model. Is an NDA required to share a vendor Blackbox model for SC analysis? Difference in VRT detection and injection logics. Angle rotation is not addressed. Current limitation logic during unbalanced faults is not clear.
Distribution control applications such as volt/var optimization, network reconfiguration, and distribution automation require accurate knowledge of the distribution system state. The lack of sufficient sensors on the primary side of distribution networks often limits the accuracy of the control decisions by these applications. The deployment of advanced metering infrastructure (AMI) provides utilities an opportunity to translate the AMI data on the secondary onto the primary so that it can be used as pseudo-measurements to augment the limited existing measurements on the primary. This paper develops an approach for estimating service transformer primary-side voltages by using limited secondary-side AMI measurements. The estimated primary-side voltages can be used by utilities as pseudo-measurements for distribution control applications. The detailed secondary model topology, which is an essential input data for many existing algorithms, is not required for the proposed method. The performance of the proposed method is validated by using AMI measurements from the field and an actual distribution feeder model of San Diego Gas & Electric Company.
The ability to predict the likelihood of impurity incorporation and their electronic energy levels in semiconductors is crucial for controlling its conductivity, and thus the semiconductor's performance in solar cells, photodiodes, and optoelectronics. The difficulty and expense of experimental and computational determination of impurity levels makes a data-driven machine learning approach appropriate. In this work, we show that a density functional theory-generated dataset of impurities in Cd-based chalcogenides CdTe, CdSe, and CdS can lead to accurate and generalizable predictive models of defect properties. By converting any semiconductor + impurity system into a set of numerical descriptors, regression models are developed for the impurity formation enthalpy and charge transition levels. These regression models can subsequently predict impurity properties in mixed anion CdX compounds (where X is a combination of Te, Se and S) fairly accurately, proving that although trained only on the end points, they are applicable to intermediate compositions. We make machine-learned predictions of the Fermi-level-dependent formation energies of hundreds of possible impurities in 5 chalcogenide compounds, and we suggest a list of impurities which can shift the equilibrium Fermi level in the semiconductor as determined by the dominant intrinsic defects. Machine learning predictions for the dominating impurities compare well with DFT predictions, revealing the power of machine-learned models in the quick screening of impurities likely to affect the optoelectronic behavior of semiconductors.
Memory design space exploration methods study memory systems’ performances and limitations before implementation. The computer memory design space has grown exponentially because of the enormous growth of memory types, memory controllers, and application software. Computer simulators are commonly used for memory design space exploration. However, complex memory simulations take an enormous amount of time. Hence, in this paper, we proposed a machine learning-based design space exploration method for dynamic random-access memory and non-volatile memory systems. We applied our method to the CosmoGAN and LeNet applications to predict the following six memory response parameters: (i) bandwidth, (ii) power, (iii) average latency, (iv) average total latency, (v) memory reads, and (vi) memory writes. Our experimental results show that machine learning models can predict memory response parameter values faster than simulations. We used support vector machine, random forest, and gradient boosting machine learning models. We observed that the support vector machine provides better performance for bandwidth, average latency, and average total latency. The random forest model works better for memory reads and writes. The gradient boosting model provides superior prediction performance for power. We provide a detailed discussion on learning curve characteristics, error analysis, and memory type recommendation.
In 2019, U.S. Department of Energy (DOE) contractor CH2M Hill Plateau Remediation Company (CHPRC) completed structurally stabilizing the Plutonium Uranium Extraction Plant (Purex) waste storage Tunnel 2 with engineered grout. This action significantly reduced risk to Hanford workers, the surrounding community and environment by preventing future collapse of the tunnel and possible release of radioactive materials. Decades of plutonium production at the Purex facility required construction of two large underground tunnels for storage of highly contaminated plutonium processing equipment that had failed during the operation of Purex and other facilities across the Hanford Site. Following the partial collapse of Purex Tunnel 1 on May 9, 2017, DOE and CHPRC took immediate action to safely stabilize the collapsed section of the tunnel. The hole in the tunnel was filled the next day as a temporary stabilization measure, and no contamination was detected as a result of the partial collapse of the tunnel. Workers then stabilized Tunnel 1 in its entirety by filling with engineered grout, which also encapsulated the radioactive equipment and railcars within the tunnel. Grouting eliminated the risk of further collapse and stabilized the tunnel contents, all while not precluding future remedial actions or final closure decisions. While conducting an integrity analysis of Tunnel 1 by using construction drawings, photos and other historical documents, project teams also analyzed the much larger Purex Tunnel 2. The analysis found Tunnel 2 did not meet current codes for structural integrity, and it may not be able to bear the weight of the soil above the tunnel. Therefore, it was determined that Tunnel 2 was also at risk of collapse. DoE's independent panel of experts determined grouting to be the best choice for stabilizing Tunnel 2 because it provides the highest level of stability and protection, and does not preclude future remedial actions. DOE authorized CHPRC to commence work and the Purex Tunnel 2 Structural Stabilization Project was launched in December of 2017. The paper will discuss the process of recognizing the following critical aspects and risks associated with stabilizing Tunnel 2, and the actions taken by the project team to mitigate these risks, including: - Tunnel investigation and data collection to assist with future remediation; - On-site batching and mock-ups to provide the large quantity of grout required for stabilizing the tunnel; - Workforce involvement and innovative actions that mitigated the project's most significant potential hazards, including traffic control; - Project team high level of execution translated into successful project execution. Sharing lessons learned from this critical risk-reduction effort can enhance the safety and efficiency of other complex and hazardous projects across the DOE complex. (authors)
Attack Chain Estimator (ACE) Application Overview The Attack Chain Estimator (ACE) Application is a sophisticated tool designed for the ingestion, classification, sequencing, and enrichment of cybersecurity threat reports. This application leverages advanced machine learning models and extensive historical data to provide comprehensive insights into cyber threats, specifically targeting Industrial Control Systems (ICS). Purpose The primary functions of the ACE Application include: Ingestion of Cybersecurity Threat Reporting: Capable of ingesting text-based threat reports in markdown or text file format. Supports ingestion of structured data from other sources in STIX/JSON format. Classification of Report’s Text-Based Events: Utilizes a DeBERTa classifier, specifically trained on cybersecurity data, to map the events to MITRE ATT&CK for ICS Tactics and Techniques. Classification is performed using multiple Jupyter notebooks and machine learning workflows hosted as FastAPI microservices: regex_data deberta_base_35_train_hft_classifier_mlflow.ipynb hft_regex_classifier_mlflow.ipynb param_train_hft_classifier_mlflow.ipynb regex_tactic_tech.ipynb Ordering of Tactics, Techniques, and Observable Events: Sequences the identified tactics, techniques, and events to form a coherent attack chain. Enrichment with Historical Attack Chain Details: Enhances the attack chain with details from historical attacks using a Markov model developed from CyOTE Precursor Analysis Report data. The Markov model is available as a FastAPI endpoint for seamless integration. Enrichment with Adversary Emulation Capabilities Data: Integrates adversary emulation capabilities data using MITRE Caldera for OT adversary abilities UUIDs. Export of Output Files: Provides options to export the enriched attack chain in JSON or CSV formats. Routing of Output to Other Applications: Facilitates routing of output to various platforms and applications, including: Threat Intelligence Platforms COREII Scout for Threat Intelligence Analysis COREII Modeling and Simulation for Adversary Emulation Technical Description The ACE Application is an advanced cybersecurity tool designed to provide detailed threat analysis and sequence generation. It is built on a robust architecture that integrates natural language processing, machine learning, and historical data modeling. Key Components: Data Ingestion Module: Handles the input of threat reports and data from various formats, ensuring flexibility in data sources. Classification Engine: Employs DeBERTa-based classifiers hosted as FastAPI microservices to analyze and classify threat report events in accordance with the MITRE ATT&CK framework for ICS. Sequence Generator: Orders the classified events into a logical attack chain, providing clear insight into the sequence of tactics and techniques used in the threat. Enrichment Engine: Integrates historical data and adversary emulation capabilities to enhance the attack chain with valuable context and additional details. The historical data enrichment is powered by a Markov model, which is available as a FastAPI endpoint. Export and Routing Module: Facilitates the export of the enriched attack chain in multiple formats and routes the output to designated applications for further analysis or emulation.
System and method for learning an intake oxygen concentration of an engine. In one embodiment, the system includes an intake oxygen sensor and an electronic control unit. The intake oxygen sensor is configured to measure an oxygen concentration of intake air and output a signal indicative of an intake oxygen concentration value. The electronic control unit is configured to receive the signal indicative of the intake oxygen concentration value, determine whether fuel is being injected into an engine, determine whether the engine is operating in an over-run condition, control a valve to sweep an air pressure of an intake path of the engine across a plurality of air pressures, and store information indicative of intake oxygen concentration values across the plurality of air pressures to learn the intake oxygen concentration of the engine.
This report provides a historical overview and lessons learned associated with the SRS sludge batch (SB) qualification and processing programs. The report covers the framework of the requirements for waste form acceptance, the DWPF Glass Product Control Program (GPCP), waste feed acceptance, examples of how the program complies with the specifications, an overview of the Startup Program, and a summary of continuous improvements and lessons learned. The report includes a bibliography of previous reports and briefings on the topic.
Advanced nuclear reactors are a key part of the future of nuclear energy both in the United States and globally. They offer unique benefits for various energy-demanding applications, including use in remote locations, compact size, modular manufacturing, remote monitoring, low and/or variable power rating operation, and reliance on novel technologies to enhance operational safety. To achieve economic feasibility, advanced reactors must significantly reduce their workforces in comparison with the current fleet. Achieving this reduction will occur through reducing staff workloads using technology to achieve autonomous or semi-autonomous operations, demonstrated by comprehensive testing and validation activities. These operations will require both software and hardware platforms during the design and testing phases. While simulations are useful during the design phase, their performance can significantly deviate during actual deployment on hardware. This report presents the outcomes of a collaborative technical initiative between the U.S. Department of Energy (DOE) Microreactor Program (MRP) and Advanced Sensors and Instrumentation (ASI) Program. The collaboration utilized the Microreactor Automated Control System (MACS) hardware platform to bridge the gap between theoretical reactor design and actual startup and control operations. Two key use cases were investigated: facilitating the startup testing period and demonstrating supervisory control. The first use case details the key Microreactor Applications Research Validation and Evaluation (MARVEL) reactor startup physics testing activities conducted using the MACS platform. These activities included drum worth measurements, shutdown margin assessment, temperature feedback analysis, and scram time evaluation, as well as unique testing that would apply to the MARVEL reactor to demonstrate the testing methodologies in a low-risk environment. The MACS platform, serving as a surrogate representation of the MARVEL reactor, proved instrumental in performing these tests. The exercise revealed aspects that led to optimized processes, refined hardware design, and enhanced base software capabilities. By maturing methods and technologies in this manner, the initiative promises to reduce wasted time in the actual on-site reactor deployment effort, thereby saving significant time and resources. The second use case focuses on the development and implementation of supervisory control methods aimed at managing core tilt, which can result from asymmetrical operations or manufacturing imperfections in fuel rods or reactivity control devices. A key objective was to assess and compare the use of artificial intelligence (AI) for supervisory control. The effort aimed to define the role of supervisory control to enhance performance without risking control instability. This effort explored three distinct approaches: rules-based (RB) methods, optimization techniques, and reinforcement learning (RL) algorithms. Each approach was evaluated for its ease of implementation, its usability, and its effectiveness in responding to asymmetries in neutron flux. Comparative analysis of these approaches provided valuable insights into their applicability and effectiveness, offering a robust framework for advanced reactor operations. Together, these two use cases highlight the potential of hardware test beds to help streamline the design, operation, and control of advanced nuclear reactors. This collaborative effort underscores the importance of continued innovation and experimentation in achieving the next generation of safe, reliable, and economically viable nuclear energy solutions.
Part 1. Design of Biomimetic Catalysts Based on Amphiphilic Systems The overall objective of our research is to create biomimetic catalysts from amphiphilic molecules. More specifically, we aim to create supramolecular systems that can be used to control the microenvironment around a catalytic center in a biomimetic fashion and apply the learning to construct supramolecular catalysts with novel functions found in enzymatic catalysts. We have prepared synthetic molecules (i.e., foldamers) that could fold into helical structures with nanometer-sized internal hydrophilic cavities. Cavities of this size are typically observed only in the tertiary and quaternary structures of proteins but were formed in our foldamer prepared in just a few steps from the monomer. Similar to many proteins, our foldamers displayed cooperativity in the folding/unfolding equilibrium and followed a two-state conformational transition. In addition, their conformational change could be triggered by solvent polarity, pH, or presence of metal ions and certain organic molecules. We studied their environmentally dependent conformational changes in solutions, surfactant micelles, and lipid bilayer membranes. Unlike conventional rigid supramolecular host, a foldamer undergoes conformational change during guest binding. Our study in the molecular recognition of an oligocholate host yielded some extremely exciting results. Cooperativity between host conformation and host–guest interactions was found to “magnify” weak binding interactions. In other words, since binding affinity is determined by the overall change of free energy during the binding, guest-induced conformational change of the host, whether near or far from the binding site, affects the binding. This study has strong implications in catalysis because enzymes have been hypothesized to harvest similar intramolecular forces to strengthen their binding with the transition state of an enzyme-catalyzed reaction. The supramolecular and amphiphilic principles used in the foldamer catalysts were extended to a few other systems, particularly to interfacially cross-linked reverse micelles and micelles. These features enabled unusual catalytic features such as basic/nucleophilic catalysis under acidic conditions. We were able to create highly active metal nanoclusters catalysts whose local environment could be tuned by the organic framework. We were even able to create a “catalytic nanomachine” that grabs the substrate to the encapsulated Au clusters, which efficiently convert the substrate to the product that is rapidly ejected due to its different binding properties. Our research has important impacts on fundamental and applied energy-related sciences. On the fundamental level, it tests important biocatalytic principles on relatively simple synthetic systems and is expected to afford deeper understanding of biological catalysis. On the practical level, the research is anticipated to lead to “smart” catalysts and open up exiting applications in chemical analysis, reaction control, and materials synthesis. Part 2. Electrochemical Reduction of CO₂ The primary objective of our research involving the electrochemical reduction of carbon dioxide is to apply a multidisciplinary approach toward developing a greater understanding of the problem of efficiently converting CO₂ to hydrocarbons through electrochemical routes. Our goal is to provide a better understanding of the principles that underlie the electrocatalytic reduction of CO₂ at electrode surfaces and the molecular pathways that lead to desired compounds. This understanding is essential for the design and development of new catalytic materials for the selective production of renewable feedstocks. The electrochemical reduction of CO₂ involves the formation of various reaction products and adsorbed intermediates whose distribution depends upon the nature of the electrode material and the electrochemical conditions, including applied potential, solvent, and electrolyte, used during reduction. Our efforts are focused on developing a detailed picture of how these various parameters impact the reaction pathway for CO₂ reduction and to tune this reaction to favor the formation of specific products by careful selection of electrode materials and reaction conditions. Our experimental plan includes a detailed examination of the electrode material as well as a quantitative evaluation of the reaction conditions using several in-situ analysis tools that give a direct measure of adsorbed and solution species that appear during CO₂ reduction. During the first year of funding for this aspect of the project, we have developed an on-line electrochemical measurement tool utilizing membrane-introduction mass spectrometry to detect reactions products directly during the electrochemical reduction of CO₂on various sputtered electrode materials. In addition, we have examined the role of a selection of different electrode compositions (including single component and binary electrodes) on product distributions.
As electricity grids employ greater fractions of renewable energy, which introduce additional variability and uncertainty in the net load, balancing electrical load and generation becomes more challenging. This paper reviews the literature documenting physical simulations and real systems that employ load control (LC) for frequency response and other grid services, which balance net load on the grid and prevent unwanted frequency excursions. Apart from academic and simulation studies, few sources exist on large-scale laboratory hardware testing or actual real-world systems that employ LC for frequency response, and we review them here. Four types of systems that we consider are: 1) Laboratory-based LC experiments, 2) Isolated microgrids that employ LC, 3) Larger grids that employ LC and 4) vehicle-to-grid (V2G) technology, using electric vehicles (EVs). In general, these systems have successfully used LC to meet their objectives, which are often keeping grid frequency within a required band. However, LC struggled to balance grid frequency in an isolated system powered by a single wind turbine, and V2G technology requires refinement in communication and control to provide optimal regulation that adheres to industry standards. As LC grows in the energy industry, we have three main recommendations: 1) encouraging system operators who use LC to publish system characteristics and lessons learned; 2) transitioning more LC theoretical/simulated systems to physical experiments, and physical experiments to real-world pilot systems; 3) demonstrating load control to support isolated, high-wind-contribution systems.