Search NASA⌕ Search

SEARCH · Search NASA

Results for “Paradigm”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22

Understanding the efficacy of concentrated interstitial carbon in enhancing the pitting corrosion resistance of stainless steel

By introducing a high fraction of interstitial carbon through low temperature carburization, the pitting corrosion resistance of austenitic stainless steel can be significantly improved. Previous work attributed this to strengthening of the passive film by carbon. Here, however, we present a paradigm shift by showing that carbon actually weakens the passive film but strengthens the bonding within the alloy by forming localized (covalent) bonds with the metal atoms (Fe, Cr, Ni). Accordingly, the enhancement in pitting resistance is a result of carbon reducing the metal dissolution rate in a local pit environment by many orders of magnitude, which extremely decreases the growth stability of a pit and prevents it from transitioning into stable growth. Electronic structure calculations confirm that carbon bonds to the metal atoms and that the metal–carbon bonds are 1.4 to 2.0 times stronger than the metal–metal bonds. These results suggest a new strategy for the design of corrosion resistant alloys, namely surface engineering by infusion of concentrated interstitials that form strong bonds with the matrix atoms.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Irradiation damage reduces alloy corrosion rate via oxide space charge compensation effects

Radiation effects in materials often compound and accelerate other detrimental phenomena such as embrittlement, oxidation and creep. However, irradiation can also decrease the oxidation rate, for instance with ZrNb alloys nuclear fuel cladding. In this study, we rationalize this observation on Zr-0.5Nb alloy by introducing a mechanism based on oxide space charge modification, resulting from irradiation enhanced Nb clustering. Here, this mechanism is investigated using a multiscale approach: from the macroscale, to determine post-irradiation oxidation kinetics, to the atomic scale, using in-situ atom probe tomography sample oxidation, to observe elemental solute redistribution across the oxide/metal interface. The mechanism is further supported by high resolution transmission electron microscopy characterization and density functional theory calculations. A point defect model is proposed to account for oxide space charge effects and their changes under irradiation. This integrated, multiscale experimental and modeling approach challenges the current paradigm on irradiation effects and how they can potentially improve materials performance in extreme environments.

36 MATERIALS SCIENCE↗

Analytical gradient-based optimization of CALPHAD model parameters

The calibration of CALPHAD (CALculation of PHAse Diagrams) models involves the solution of a very challenging high-dimensional multiobjective optimization problem. Traditional approaches to parameter fitting predominantly rely on gradient-free methods, which while robust, are computationally inefficient and often scale poorly with model complexity. In this work, we introduce and demonstrate a generalizable framework for analytic gradient-based optimization of the parameters of the CALPHAD model enabled by the recently formalized Jansson derivative technique. This method allows for efficient evaluation of gradients of thermodynamic properties at equilibrium with respect to model parameters, even in the presence of arbitrarily complex internal degrees of freedom. Leveraging these semi-analytic gradients, we employ the conjugate gradient (CG) method to optimize thermodynamic model parameters for four binary alloy systems: Cu-Mg, Fe-Ni, Cr-Ni, and Cr-Fe. Across all systems, CG achieves comparable or superior optimality relative to Bayesian ensemble Markov Chain Monte Carlo (MCMC) with improvements in computational efficiency ranging from one to three orders of magnitude. Furthermore, our results establish a new paradigm for CALPHAD assessments in which high fidelity data-rich model calibration becomes tractable using deterministic gradient-informed algorithms.

CALPHAD↗

Large-scale additive manufacturing of self-heating molds

Large-scale material extrusion additive manufacturing technology is becoming the new mainstream technology for scaled-up composite mold and die applications. This paradigm shift in composite processing technology is primarily driven by out-of-autoclave tooling applications, in which fiber reinforced composite molds with scaled-up sizes and embedded heating elements are attractive. The present research describes the design, manufacturing, and testing of self-heating composite molds fabricated via a large-scale pellet extrusion 3D printing machine with an integrated wire co-extrusion tool. Polycarbonate (PC) composites reinforced with carbon fiber (PC/CF; 20 wt.%) and glass fiber (PC/GF; 20 wt.%) were used to fabricate mold parts. Joule heating thermal test results showed that uniform temperatures (~100 °C) were achieved for both PC/CF and PC/GF mold surfaces, using a custom-made feedback control power supply and infrared thermography. Mechanical characterizations, including tensile and flexural testing were performed on the wire-embedded and un-wired PC/CF and PC/GF base specimens to investigate the impact of the fiber reinforcement as well as the embedded wires. In the direction of extrusion, the ultimate tensile stress of PC/CF was 105 MPa, and that of PC/GF was 73 MPa, while the neat PC value was 64 MPa. Inner-bead voids and interfacial gaps were observed and characterized via optical and scanning electron microscopy. The embedded wires and inner bead impacted the mechanical properties of the composites. Furthermore, the stiffness of the wire-embedded mold was still satisfactory, proving that the technology can be used to fabricate additively manufactured out-of-oven/autoclave molds.

36 MATERIALS SCIENCE↗

Scalable in situ non-destructive evaluation of additively manufactured components using process monitoring, sensor fusion, and machine learning

Laser Powder Bed Fusion (L-PBF) Additive Manufacturing (AM) is among the metal 3D printing technologies most broadly adopted by the manufacturing industry. However, the current industry qualification paradigm for critical-application L-PBF parts relies heavily on expensive non-destructive inspection techniques, which significantly limits the use-cases of L-PBF. In situ monitoring of the process promises a less expensive alternative to ex situ testing, but existing sensor technologies and data analysis techniques struggle to detect sub-surface flaws (e.g., porosity and cracking) on production-scale L-PBF printers. In this work, an in situ NDE (INDE) system was engineered to detect subsurface flaws detected in X-Ray Computed Tomography (XCT) directly from process monitoring data. A multilayer, multimodal data input allowed the INDE system to detect numerous subsurface flaws in the size range of 200–1000µm using a novel human-in-the-loop annotation procedure. Furthermore, a framework was established for generating probability-of-detection (POD) and probability-of-false-alarm (PFA) curves compliant with NDE standards by systematically comparing instances of detected subsurface flaws to post-build XCT data. Here, we also introduce for the first time in the AM in situ sensing literature the a 90/95 – the flaw size corresponding to a 90% detection rate on the lower 95% confidence interval of the POD curve. The INDE system successfully demonstrated POD capabilities commensurate with traditional NDE methods. Traditional ML performance metrics were also shown to be inadequate for assessing the ability of the INDE system’s flaw detection performance. It is the hope of the authors that future studies will adopt the POD and PFA approach outlined here to provide better insight into the utility of process monitoring for AM.

36 MATERIALS SCIENCE↗

Hydrologic investigations of radar-rainfall error propagation to rainfall-runoff model hydrographs

Rainfall is arguably the most important yet most variable input for rainfall-runoff hydrologic models. In this study, the authors search for the characteristics of radar-rainfall estimates that are most important for skillful streamflow predictions. They perform comprehensive hydrologic investigations of radar-rainfall characteristics, including spatiotemporal resolution, radar range visibility, statistical characterization of rainfall variability, all vis-a-vis basin characteristics such as size and river network topology. Since the true rainfall fields are unknown, the authors exploit a paradigm of using two independently constructed radar-rainfall products i.e., Multi-Radar Multi-Sensor and IFC-ZR used operationally by the Iowa Flood Center (IFC). Using the distributed hydrologic model called the Hillslope-Link Model for the domain of the state of Iowa, they evaluate streamflow prediction at 140 USGS gauge stations that monitor rivers in Iowa. Through spatial and temporal rainfall aggregation experiments, the authors show that the impact of spatial and temporal resolution of rainfall is significant typically for smaller basins while starts reducing significantly for basins larger than 1,000 km 2 . Other rainfall characteristics they explored do not reveal a strong signature in the relationship of rainfall differences between the two products and hydrograph errors. However, exploring the product similarities rather than differences reveals that the basin-wide rainfall volume has the most significant effect on streamflow prediction. The results from this study are generalizable for all rainfall observing systems.

54 ENVIRONMENTAL SCIENCES↗

Making sense of uncertain nuclear data

At its core, and in an imperfect world, the scientific method relies heavily on the concept of uncertain knowledge. Understanding the nature and magnitude of our lack of knowledge on nuclear data has important ramifications on a wide range of nuclear technologies that encompass defense, energy, medicine, astrophysics, among others. Over the span of a long and fruitful carrier, Dr. Massimo Salvatores has made significant contributions to the fields of nuclear reactor physics, nuclear data, and uncertainty quantification and propagation. Advances in machine learning techniques and computational capabilities are enabling a new level of confidence in our ability to predict key nuclear metrics as well as their associated uncertainties across many applications. Here this short article reviews some of those concepts with an eye towards developing a new paradigm to provide nuclear data libraries of benefit to various applications.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Design of a supervisory control system for autonomous operation of advanced reactors

Advanced reactors to be deployed in the coming decades will face deregulated energy markets, and may adopt flexible operation to boost profitability. To aid in the transition from baseload to flexible operation paradigm, autonomous operation is sought. This work focuses on the control aspect of autonomous operation. Specifically, a hierarchical control system is designed to support constraint enforcement during routine operational transients. Within the system, data-driven modeling, physics-based state observation, and classical control algorithms are integrated to provide an adaptable and robust solution. A 320 MW Fluoride-cooled High-temperature Pebble-bed Reactor is the design basis for demonstrating the proposed control system. The hierarchical control system consists of a supervisory layer and low-level layer. The supervisory layer receives requests to change the system's operating conditions (e.g., the current reactor power to meet a load -follow), and accepts or rejects them based on constraints that have been assigned. Constraints are issued to keep the plant within an optimal operating region. The low-level layer interfaces with the actuators of the system to fulfill requested changes, while maintaining tracking and regulation duties. Further, to accept requests at the supervisory layer, the Reference Governor algorithm was adopted. To model the dynamics of the reactor, a system identification algorithm, Dynamic Mode Decomposition, was utilized. To estimate the evolution of process variables that cannot be directly measured (e.g., the propagation of delayed neutron precursors), the Unscented Kalman Filter, incorporating a nonlinear model of nuclear dynamics, was adopted. The composition of these algorithms led to a numerical demonstration of constraint enforcement during a 40% power drop transient (at a rate of 5 %/min). Uncontrolled secondary-side temperatures were successfully constrained. Adaptability of the proposed system was demonstrated by modifying the constraint values, and enforcing them during the transient. Robustness was also demonstrated by enforcing constraints under noisy environments.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Impact of GW170817 for the nuclear physics of the EOS and the r-process

The recent LIGO/Virgo detection of gravitational radiation from GW170817 bears the clear signature of a binary neutron star merger and has validated a number of theoretical predictions. High-energy radiation in the form of a short gamma-ray burst was observed 1.7 s following the merger, and an extended optical/infrared afterglow lasting weeks was observed by dozens of observatories. Short gamma-ray bursts had been thought to be produced by mergers, and the afterglow is the predicted signal of radioactive decays from decompressing neutron-rich matter catastrophically ejected from the merging stars. The afterglow provided solid evidence that neutron star mergers are a major, if not primary, source of r-process nuclei, that half of all nuclei heavier than iron formed from the rapid capture of neutrons. Although this idea was proposed nearly 45 years ago, it was largely ignored in favor of a supernova mechanism. Over the last decade, however, evidence has been accumulating that supernovae are not the primary r-process source. GW170817 may have finally settled this question, which has been one of the thorniest problems in nuclear physics and astrophysics. This article presents my personal perspective of this paradigm shift.

79 ASTRONOMY AND ASTROPHYSICS↗

Superconductivity in dilute SrTiO 3 : A review

Doped SrTiO 3 , one of the most dilute bulk systems to display superconductivity, is perhaps the first example of an unconventional superconductor, as it does not fit into the standard BCS paradigm. More than five decades of research has revealed a rich temperature-carrier concentration phase diagram that showcases a superconducting dome, proximity to a putative quantum critical point, Lifshitz transitions, a multi-gap pairing state and unusual normal-state transport properties. Research has also extended beyond bulk SrTiO 3 , ushering the new field of SrTiO 3 -based heterostructures. Because many of these themes are also featured in other quantum materials of contemporary interest, recent years have seen renewed interest in SrTiO 3 . Here, we review the challenges and recent progress in elucidating the superconducting state of this model system. At the same time that its extreme dilution requires to revisit several of the approximations that constitute the successful Migdal–Eliashberg description of electron–phonon superconductivity, including the suppression of the Coulomb repulsion via the Tolmachev–Anderson–Morel mechanism, it opens interesting routes for alternative pairing mechanisms whose applicability remains under debate. For instance, pairing mechanisms involving longitudinal optical phonons have to overcome the hurdles created by the anti-adiabatic nature of the pairing interaction, whereas mechanisms that rely on the soft transverse optical phonons associated with incipient ferroelectricity face challenges related to the nature of the electron–phonon coupling. Proposals in which pairing is mediated by plasmons or promoted locally by defects are also discussed. In conclusion, we finish by surveying the existing evidence for multi-band superconductivity and outlining promising directions that can potentially shed new light on the rich problem of superconductivity in SrTiO 3 .

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

Higher-rank tensor field theory of non-abelian fracton and embeddon

We formulate a new class of tensor gauge field theories in any dimension that is a hybrid class between symmetric higher-rank tensor gauge theory (i.e., higher-spin gauge theory) and anti-symmetric tensor topological field theory. Our theory describes a mixed unitary phase interplaying between gapless and gapped topological order phases (which can live with or without Euclidean, Poincaré, or anisotropic symmetry, at least in ultraviolet high or intermediate energy field theory, but not yet to a lattice cutoff scale). The “gauge structure” can be compact, continuous, abelian or non-abelian. Our theory sits outside the paradigm of Maxwell electromagnetic theory in 1865 and Yang–Mills isospin/color theory in 1954. We discuss its local gauge transformation in terms of the ungauged vector-like or tensor-like higher-moment global symmetry. The non-abelian gauge structure is caused by gauging the non-commutative symmetries: a higher-moment symmetry and a charge conjugation (particle–hole) symmetry. Vector global symmetries along time direction may exhibit time crystals. We explore the relation of these long-range entangled matters to a non-abelian generalization of Fracton order in condensed matter, a field theory formulation of foliation, the spacetime embedding and Embeddon that we newly introduce, and possible fundamental physics applications to dark matter or dark energy.

36 MATERIALS SCIENCE↗

Information-theoretic aspects of Werner states

Highlights: • The rich structures of Werner states are reviewed. • Werner states are characterized in terms of quantum uncertainty. • Information-theoretic aspects of Werner states are revealed. In a seminal study of quantum states with Einstein–Podolsky–Rosen correlations (entanglement) admitting a hidden-variable model (Werner, 1989), Werner introduced the dichotomy of entanglement/separability and devised a family of highly symmetric states, now termed the Werner states, some of which exhibit entanglement but no Bell nonlocality. It turns out that the Werner states have a rich structure of correlations and constitute a paradigm which has played an innovative role in both theoretical and experimental explorations of quantum information. Given the theoretical significance and wide applications of the Werner states, here we first give a concise review of information contents of the Werner states, and then present an information-theoretic characterization of them in terms of the Wigner–Yanase skew information: The Werner states are identified as the states with the minimum quantum uncertainty with respect to a natural family of observables (i.e., the generators of the diagonal unitary group). For this purpose, we introduce a measure of quantum uncertainty which is of independent interest in studying asymmetry, coherence, and uncertainty, and reveal its fundamental properties. We further identify the Bell triplet states as the opposite states of the Werner states in the sense that they have the maximal amount of quantum uncertainty. Analogously, we provide a similar characterization of the isotropic states as the minimum quantum uncertainty states with respect to a closely related family of operators.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Physics of the Inverted Harmonic Oscillator: From the lowest Landau level to event horizons

In this work, we present the inverted harmonic oscillator (IHO) Hamiltonian as a paradigm to understand the quantum mechanics of scattering and time-decay in a diverse set of physical systems. As one of the generators of area preserving transformations, the IHO Hamiltonian can be studied as a dilatation generator, squeeze generator, a Lorentz boost generator, or a scattering potential. In establishing these different forms, we demonstrate the physics of the IHO that underlies phenomena as disparate as the Hawking–Unruh effect and scattering in the lowest Landau level (LLL) in quantum Hall systems. We derive the emergence of the IHO Hamiltonian in the LLL in a gauge invariant way and show its exact parallels with the Rindler Hamiltonian that describes quantum mechanics near event horizons. This approach of studying distinct physical systems with symmetries described by isomorphic Lie algebras through the emergent IHO Hamiltonian enables us to reinterpret geometric response in the lowest Landau level in terms of relativistic effects such as Wigner rotation. Further, the analytic scattering matrix of the IHO points to the existence of quasinormal modes (QNMs) in the spectrum, which have quantized time-decay rates. We present a way to access these QNMs through wave packet scattering, thus proposing a novel effect in quantum Hall point contact geometries that parallels those found in black holes.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Hydrogenation and C-S bond activation pathways in thiophene and tetrahydrothiophene reactions on sulfur-passivated surfaces of Ru, Pt, and Re nanoparticles

Thiophene-H2 reactions proceed via sulfur removal and hydrogenation routes on dispersed metal nanoparticles that become decorated by refractory S-adlayers during catalysis. The identity and kinetic relevance of the required elementary steps are described here based on rates measured at S-chemical potentials set by H2S/H2 ratios similar to those prevalent during practical catalysis on Re, ReSx, Ru, and Pt catalysts. Free energies for S adatom formation (from H2S decomposition and H2 evolution) are strongly exothermic (< -50 kJ mol-1 on Pt(111) and < -150 kJ mol-1 on Re and Ru(0001)), but strong repulsions between S adatoms cause adsorption free energies to increase significantly with coverage on all three surfaces, preventing complete monolayer formation. These adlayers, composed of unreactive S-atoms (S') that cover 1/3–2/3 ML leave residual interstitial spaces (*) that bind S-atoms (S*), intermediates, and transition states reversibly, as required for catalytic turnovers. The number and binding properties of these interstices depend on the identity and chemical state of the nanoparticle bulk phase, which influences S'-binding and coverages and cause large differences in direct desulfurization and hydrogenation turnover rates (per exposed metal atom) on dispersed Re, ReSx, Ru, and Pt. The identity and kinetic relevance of elementary steps for desulfurization (to C4¬ hydrocarbons) and hydrogenation (to tetrahydrothiophene; THT) are similar among these catalysts; they involve the kinetically-relevant formation of a thiophene-derived intermediate (monohydrothiophene on Re and ReSx; dihydrothiophene on Ru and Pt) that either cleaves its C-S bond or “over-hydrogenates” to THT in one surface sojourn. THT then undergoes C-S bond cleavage in secondary reactions that correct such over-hydrogenation to form the more unsaturated species that cleave C-S bonds. THT/C4 product ratios are insensitive to H2S/H2 ratios and thiophene pressure, even though active interstitial spaces are covered by kinetically-detectable coverages of S* and thiophene; therefore, primary and secondary reactions must involve the same active surfaces. The observed increase in THT/C4 ratios with H2 pressure shows that THT formation transition states involve a larger number of H-atoms than for C-S cleavage. The requirement for bound species with intermediate unsaturation (between THT and thiophene) for C-S bond cleavage is reminiscent of the H-shuttling required in C-C and C-O hydrogenolysis, reactions that involve the partial dehydrogenation of alkanes and alkanols, respectively, to weaken such bonds and to increase the formation entropy of the relevant transition states via the evolution of H2(g). These mechanistic details challenge prevalent paradigms about different site requirements for hydrogenation and desulfurization pathways and about how metal-sulfur bond energies act as descriptors of reactivity; in fact, such binding energies merely act to define the refractory S-adlayers that enable the formation of weakly-binding interstices that reversibly bind intermediates and transition states, thus allowing catalytic turnovers.

Yik, Edwin↗

The importance of maldistribution matching for thermal performance of compact heat exchangers

Compact heat exchangers have gained increased attention in recent years, particularly in demanding applications where high temperatures, high pressures, and/or high power densities are required. For decades, the heat exchanger (HX) community believes that flow maldistribution is a key factor for HX effectiveness, that is, reducing the degree of flow maldistribution (MALD) can help increase the HX effectiveness. Therefore, significant efforts have been devoted in the past to optimizing the header geometry to minimize flow maldistribution. This work was initially motivated by this, and the original goal was to figure out a HX header design with the lowest maldistribution. However, by systematically constructing a comprehensive maldistribution matrix, the analysis revealed that the HX effectiveness is not actually determined by the MALD, but instead dominated by the degree of maldistribution mismatch (MISM). This conclusion was also theoretically generalized, which indicated that matching of the local heat capacity rate is key for achieving maximum performance. The MISM provides a local means of tracking this information, while the MALD only provides a global approximation of the maldistribution itself. With this new perspective, flow maldistribution needs not necessarily be avoided, but instead matched between two fluid streams, to improve the HX performance. We demonstrated that by carefully designing the header geometry to match the velocity profiles of the two fluids in a 2 MW PCHE with molten salt and supercritical carbon dioxide (sCO2) as the heat transfer fluids, the HX could achieve a higher effectiveness even when the maldistribution increased. Finally, a technoeconomic study using a CSP system as an example revealed that the use of this new HX design paradigm could result in CSP capital cost savings as large as 16.6%.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Robust trajectory-constrained frequency control for microgrids considering model linearization error

Grid supportive modes integrated within inverter-based resources can improve the frequency response of renewable-rich microgrids. The synthesis of grid supportive modes to guarantee frequency trajectory constraints under a predefined disturbance set is challenging but essential. To tackle this challenge, a numerical optimal control (NOC)-based control synthesis methodology is proposed. Without loss of generality, a wind-diesel fed microgrid is studied, where we aim to design grid supportive functions in the wind turbine. In the control design, linearized models are used, and the linearization-induced errors are quantitatively analyzed by reachability and interval arithmetics and represented in the form of interval uncertainties. Then, the NOC problem can be formulated into a robust mixed-integer linear program. The control structure is strategically configured into two levels to realize online deployment. The proposed control is verified on the modified 33-node microgrid with a full-order three-phase nonlinear model in Simulink. In conclusion, the simulation results show the effectiveness of the proposed control paradigm and the necessity of considering linearization-induced uncertainty.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Heteroepitaxial registry and band structures at the polar-to-polar STO/ZnO($000\bar{1}$) interfaces

The interface between ZnO and SrTiO 3 (STO) provides a paradigm for combining perovskite oxides and wurtzite-structure semiconductors. However, the heteroepitaxial and energy band structures of the polar-to-polar STO/ZnO($000\bar{1}$) interfaces has rarely been studied. In this study, it is shown that the STO films prepared on the ZnO($000\bar{1}$) substrate by pulsed laser deposition possess [011]STO and [111]STO azimuth orientations, which exhibit three- and two-fold rotation domains respectively, as demonstrated by X-ray diffraction and transmission electron microscopy. The band structures of the STO(011)/ZnO($000\bar{1}$) and STO(111)/ZnO($000\bar{1}$) interfaces are bending downward as shown in the I-V characteristic spectra and first principle calculations. These results are different from the single orientation ZnO films grown on STO-(011) and -(111) substrate, as reported in previous literatures, showing the heteroepitaxial asymmetry between STO/ZnO and ZnO/STO interfaces. This work presents an approach towards the physical modeling of combinations between perovskite oxides and wurtzite-structure semiconductors.

77 NANOSCIENCE AND NANOTECHNOLOGY↗

Reinforcement Learning for feedback-enabled cyber resilience

The rapid growth in the number of devices and their connectivity has enlarged the attack surface and made cyber systems more vulnerable. As attackers become increasingly sophisticated and resourceful, mere reliance on traditional cyber protection, such as intrusion detection, firewalls, and encryption, is insufficient to secure the cyber systems. Cyber resilience provides a new security paradigm that complements inadequate protection with resilience mechanisms. A Cyber-Resilient Mechanism (CRM) adapts to the known or zero-day threats and uncertainties in real-time and strategically responds to them to maintain the critical functions of the cyber systems in the event of successful attacks. Feedback architectures play a pivotal role in enabling the online sensing, reasoning, and actuation process of the CRM. Reinforcement Learning (RL) is an important gathering of algorithms that epitomize the feedback architectures for cyber resilience. It allows the CRM to provide dynamic and sequential responses to attacks with limited or without prior knowledge of the environment and the attacker. In this work, we review the literature on RL for cyber resilience and discuss the cyber-resilient defenses against three major types of vulnerabilities, i.e., posture-related, information-related, and human-related vulnerabilities. Here we introduce moving target defense, defensive cyber deception, and assistive human security technologies as three application domains of CRMs to elaborate on their designs. The RL algorithms also have vulnerabilities themselves. We explain the major vulnerabilities of RL and present develop several attack models where the attacker target the information exchanged between the environment and the agent: the rewards, the state observations, and the action commands. We show that the attacker can trick the RL agent into learning a nefarious policy with minimum attacking effort. The paper introduces several defense methods to secure the RL-enabled systems from these attacks. However, there is still a lack of works that focuses on the defensive mechanisms for RL-enabled systems. Last but not least, we discuss the future challenges of RL for cyber security and resilience and emerging applications of RL-based CRMs.

97 MATHEMATICS AND COMPUTING↗