Search NASA⌕ Search

SEARCH · Search NASA

Results for “fault tolerant systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Physics-informed State-space Neural Networks for transport phenomena

This work introduces Physics -informed State -space neural network Models (PSMs), a novel solution to achieving real-time optimization, flexibility, and fault tolerance in autonomous systems, particularly in transportdominated systems such as chemical, biomedical, and power plants. Traditional data -driven methods fall short due to a lack of physical constraints like mass conservation; PSMs address this issue by training deep neural networks with sensor data and physics -informing using components' Partial Differential Equations (PDEs), resulting in a physics -constrained, end -to -end differentiable forward dynamics model. Further, through two in silico experiments - a heated channel and a cooling system loop - we demonstrate that PSMs offer a more accurate approach than a purely data -driven model. In the former experiment, PSMs demonstrated significantly lower average root -mean -square errors across test datasets compared to a purely data -driven neural network, with reductions of 44 %, 48 %, and 94 % in predicting pressure, velocity, and temperature, respectively. Beyond accuracy, PSMs demonstrate a compelling multitask capability, making them highly versatile. In this work, we showcase two: supervisory control of a nonlinear system through a sequentially updated state -space representation and the proposal of a diagnostic algorithm using residuals from each of the PDEs. The former demonstrates PSMs' ability to handle constant and time -dependent constraints, while the latter illustrates their value in system diagnostics and fault detection.

42 ENGINEERING↗

Collaborative Fault Tolerant Control of Non-Signalized Intersections for Connected and Autonomous Vehicles

With the potential of increased penetration of connected and autonomous vehicles (CAVs), intersectional signal control faces new challenges in terms of its operation and implementation. One possibility is to fully make use of the communication capabilities of CAVs so that intersectional signal control can be realized by CAVs alone – this leads to non-signalized intersectional operation for traffic networks in urban areas. In this paper, the state-of-the-art on collaborative fault tolerant control schemes for complex systems will be briefly described. This is then followed by the formulation of operational fault tolerant control that realizes the collaborative fault tolerance functionality at CAVs operational level in response to possible individual vehicle faults, where detailed modelling using vehicle movement dynamics will be described together with the construction of fast fault diagnosis and a collaborative fault tolerant control algorithm. A simple example will be given as well to demonstrate the proposed algorithm together with the discussions on other issues such as randomness of the system, communication errors and full energy consideration. These leads to several future directions of the research for the traffic flow control of non-signalized intersections with 100% penetration of CAVs.

Wang, Hong↗

Enhancing quantum utility: Simulating large-scale quantum spin chains on superconducting quantum computers

We present the quantum simulation of the frustrated quantum spin- 1 2 antiferromagnetic Heisenberg spin chain with competing nearest-neighbor ( J 1 ) and next-nearest-neighbor ( J 2 ) exchange interactions in the real superconducting quantum computer with qubits ranging up to 100. In particular, we implement the Hamiltonian with the next-nearest neighbor exchange interaction in conjunction with the nearest-neighbor interaction on IBM's superconducting quantum computer and carry out the time evolution of the spin chain by employing the first-order Trotterization. Furthermore, our implementation of the second-order Trotterization for the isotropic Heisenberg spin chain, involving only nearest-neighbor exchange interaction, enables precise measurement of the expectation values of staggered magnetization observable across a range of up to 100 qubits. Notably, in both cases, our approach results in a constant circuit depth in each Trotter step, independent of the number of qubits. Our demonstration of the accurate measurement of expectation values for the large-scale quantum system using superconducting quantum computers designates the quantum utility of these devices for investigating various properties of many-body quantum systems. This will be a stepping stone to achieving the quantum advantage over classical ones in simulating quantum systems before the fault tolerance quantum era. Published by the American Physical Society 2024

97 MATHEMATICS AND COMPUTING↗

Designing a decentralized fault-tolerant software framework for smart grids and its applications

The vision of the ‘Smart Grid’ anticipates a distributed real-time embedded system that implements various monitoring and control functions. As the reliability of the power grid is critical to modern society, the software supporting the grid must support fault tolerance and resilience of the resulting cyber-physical system. This paper describes the fault-tolerance features of a software framework called Resilient Information Architecture Platform for Smart Grid (RIAPS). The framework supports various mechanisms for fault detection and mitigation and works in concert with the applications that implement the grid-specific functions. The paper discusses the design philosophy for and the implementation of the fault tolerance features and presents an application example to show how it can be used to build highly resilient systems.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Fault-Tolerant Deep Learning Cache with Hash Ring for Load Balancing in HPC Systems

Large-scale DL on HPC systems like Frontier and Summit uses distributed node-local caching to address scalability and performance challenges. However, as these systems grow more complex, the risk of node failures increases, and current caching approaches lack fault tolerance, jeopardizing large-scale training jobs. We analyzed six months of SLURM job logs from Frontier and found that over 30% of jobs failed after an average of 75 minutes. To address this, we propose fault-tolerance strategies that recache data lost from failed nodes using a hash ring technique for balanced data recaching in the distributed node-local caching, reducing reliance on the PFS. Our extensive evaluations on Frontier showed that the hash ring-based recaching approach reduced training time by approximately 25% compared to the approach that redirects I/O to the PFS after node failures and demonstrated effective load balancing of training data across nodes.

Lee, Seoyeong↗

Report on local data recovery approaches suitable for weather and climate prediction (Deliverable 1.3) (V.1.0)

Numerical weather and climate prediction rates as one of the scientific applications whose accuracy improvements greatly depend on the growth of the available computing power. As the number of cores in top computing facilities pushes into the millions, increasing average frequency of hardware and software failures forces users to review their algorithms and systems in order to protect simulations from breakdown. This report surveys approaches for fault-tolerance in numerical algorithms and system resilience in parallel simulations from the perspective of numerical weather and climate prediction systems. A selection of existing strategies is analyzed, featuring interpolation-restart and compressed checkpointing for the numerics, in-memory checkpointing, user-level failure mitigation-based and backup-based methods for the systems. Numerical examples showcase the performance of the techniques in addressing faults, with particular emphasis on iterative solvers for linear systems, a staple of atmospheric fluid flow solvers. The potential impact of these strategies is discussed in relation to current development of numerical weather prediction algorithms and systems towards the exascale. Trade-offs between performance, efficiency and effectiveness of resiliency strategies are analyzed and some recommendations outlined for future developments.

97 MATHEMATICS AND COMPUTING↗

Demonstration of a fault impact reduction control module for wind turbines

Abstract. Traditionally, wind turbines in distributed applications make control decisions as isolated systems. They generally provide maximum power output during operation and manage internal faults with little consideration of the rest of the power system. Although fault detection and tolerance schemes are widely researched and implemented, controls to ameliorate such faults are uncommon in research and industry. The rapid shutdown of a wind turbine in a large transmission-connected wind plant will have a minimal impact on a large power system, but in a microgrid or isolated grid context the abrupt loss of a single wind turbine may cause grid instability and high stress on the system. This paper demonstrates a fault impact reduction control (FIRC) module for a wind turbine, which implements wider warning thresholds around fault thresholds. When the turbine crosses a warning threshold, the controller sends its predicted action to the grid controller, which facilitates the grid operator’s response to a potential wind turbine fault and then takes appropriate action to ameliorate the fault. Various test cases demonstrate the controller action under a variety of faults, and various scenarios demonstrate the grid benefit of an FIRC in both microgrid- and grid-connected contexts. The FIRC maximizes wind turbine generation and eases generation transition under a variety of fault scenarios. The FIRC module is easy to integrate with most existing controllers, just requiring derate capability, and can be easily modified to include the various warnings and thresholds that the user desires. This analysis is mainly performed in a MATLAB-Simulink-based research wind turbine model and is also implemented in the existing LabVIEW-based controller of the same research turbine at NREL.

17 WIND ENERGY↗

Benchmarking Variables for Checkpointing in HPC Applications

Checkpoint/Restart (C/R) is a widely used fault tolerance mechanism in converged systems of cloud, edge, and HPC. However, users often rely on their experience to determine which variables to checkpoint, as there is currently no benchmark that can provide a reference. This can result in checkpointing redundant or even incorrect variables. To address this issue, we propose a benchmark suite that includes critical variables for checkpointing, which have been manually identified, and a method for identifying those critical variables, with 20 representative HPC applications. Our method involves analyzing data dependency between variables to identify critical variables analytically. We verify the identified variables' correctness with a widely used C/R library FTI by an ablation study. With our benchmark suite and data dependency analysis, HPC practitioners now have a reference for identifying checkpointing variables and better knowledge of what kind of variables to checkpoint.

Fu, Xiang↗

Blockchain for Fault-Tolerant Grid Operations

Radial topology and vast geographic coverage make distribution systems prone to widespread power outages upon the failure of a single (or multiple) upstream component. Fault-handling algorithms depend heavily on correct estimations of the system’s state to effectively isolate the affected area and reduce the number of affected customers while maintaining operational safety. The work described here leverages the core features of distributed, consensus-based decision-making processes and the immutability of blockchain, and demonstrates their value in improving fault-tolerant grid operations. In this work, blockchain was used to create a trusted data-sharing platform that enables independent actors to reconstruct the system state; this enables distributed resources to make intelligent decisions with limited knowledge. Although the process requires data sharing, its algorithms have been designed to limit the amount of private information that is exchanged, which helps preserve business-sensitive data and maintain customer privacy. In addition, by reducing the information that must be shared, the communication requirements are also reduced; (however, an in-depth analysis of the communication requirements is beyond the scope of this project). The proposed use cases are intended to represent a foundational basis for third parties to develop functional solutions that can eventually be deployed in the field. To further provide guidance, the envisioned use cases have incorporated design requirements that consider the blockchain characteristics and a need to limit information from surrounding resources, which preserve the assumption and the possibility that such resources could belong to different entities. This report presents a detailed design of the three use cases with the tools needed to enable the analysis being tested. The implemented gross error detection method can detect mismatches when the error exceeds 3.8 times the sensor’s rated accuracy. Detection of the circuit breaker state successfully identified the correct states across all simulation tests. A distribution-system power-flow solution in the simulator OpenDSS generally possesses a convergency tolerance of 0.01% on the voltage magnitude. The evaluation of possible reconnection using voltage magnitude—preserving the data ownership—has a voltage magnitude difference smaller than 0.001% from the OpenDSS result. The results preserving data ownership have a difference within the expected power flow tolerance with full knowledge of the system, which surpasses expectations.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Fault-tolerant grid frequency measurement algorithm during transients

A system determines the frequency of grid signals corresponding to an electrical grid in real time. The system includes a transient detector that monitors a grid signal from a voltage meter or a current meter connected to the electrical grid. The system produces, in real time and at a sampling rate, a deviation signal indicative of a periodicity of the monitored grid signal. The system determines, over one or more cycles of the monitored grid signal, a measurement signal corresponding to the deviation signal. The system determines a frequency signal that corresponds a frequency estimation of the monitored signal by applying a frequency estimation when values of the measurement signal are less than a deviation threshold and maintaining the frequency signal at a constant value when values of the measured signal equal or exceeds the deviation threshold.

Zhan, Lingwei↗

Blockchain for Fault-Tolerant Grid Operations Version 2.0

This report explores the potential of distributed ledger technology (DLT) as a transformative tool to enhance fault-tolerant operations in electrical distribution systems. Leveraging DLT's core attributes, including an immutable decentralized ledger, distributed consensus mechanisms, and state replication capabilities, this study focuses on three critical use cases. A central aspect of this research centers on the utilization of a consensus-driven ledger, providing actors within the system, such as distributed resources, with access to a reliable data repository. This empowers these actors to collaborate effectively and make informed decisions, all securely recorded on the blockchain. The first use case concentrates on data configuration, utilizing mathematical criteria---particularly, the chi-squared test for gross error detection---to identify trustworthy sensors for advanced decision-making. Building upon this foundation of trust, the second use case, topology identification, accurately determines circuit breaker states, unveiling the distribution network's topology. Ultimately, the third use case leverages this trust to execute switching actions, reconfiguring feeders and restoring power to disconnected customers after fault events. The concept of trust serves as a cornerstone in this approach, marking a departure from traditional fault location, isolation, and service restoration (FLISR) methods. Additionally, the blockchain-based architecture introduces decentralization, empowering disconnected areas to make autonomous decisions, even when communication with a central control center is disrupted. The primary contributions of this report are twofold: (1) a novel approach for evaluating distribution system voltage areas while preserving data ownership and (2) the implementation of interactions between distribution network areas using the actor model. Unlike the previous sequential approach for evaluating the area connection voltages, which required a radial network topology, this study's area model reduction enables a more versatile approach. The area model reduction addresses issues of prolonged data waiting times and multiple points of failure within the previous approach. Notably, the presented evaluation for the reduced network model area connection reveals a significant increase in the differences in voltage magnitudes. Simulation and evaluation of area agents across four distinct cases elucidate the area-level interaction behavior during a fault event. Simulations demonstrate that the proposed distributed FLISR (DFLISR) approach can successfully restore service to an affected area. Varying message delays and message loss probabilities in each simulation case underscore their impacts on restoration times, ranging from 3 min and 32 s to 6 min and 19 s. In contrast, power is not restored in an area in one of our simulation cases.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Simulation and power quality analysis of a Loose-Coupled bipolar DC microgrid in an office building

With distributed generation and battery storage technologies thriving in microgrids, the use of direct current (DC) microgrids in the building sector offers multiple advantages in energy efficiency and power quality compared with alternating current (AC) systems. This study developed a new concept of a loose-coupled bipolar DC building power system. In this work, the concept was used to design a real-world office building in Shenzhen, China. A power system model was developed to study the stability and control of the DC power system and to verify DC power quality. The design and modeling of the DC power control system is discussed in detail. The study developed a few common fault scenarios in DC building microgrids that were simulated in the MATLAB-Simulink environment to validate the design of a loose-coupled bipolar DC system. The results indicate that the proposed loose-coupled bipolar DC system schema, when implemented with proper control algorithms, can achieve good fault-tolerant performance with reliable power quality, even during disruptive system events.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

A Visual Comparison of Silent Error Propagation

High-performance computing (HPC) systems play a critical role in facilitating scientific discoveries. Their scale and complexity (e.g., the number of computational units and software stack) continue to grow as new systems are expected to process increasingly more data and reduce computing time. However, with more processing elements, the probability that these systems will experience a random bit-flip error that corrupts a program's output also increases, which is often recognized as silent data corruption. Analyzing the resiliency of HPC applications in extreme-scale computing to silent data corruption is crucial but difficult. An HPC application often contains a large number of computation units that need to be tested, and error propagation caused by error corruption is complex and difficult to interpret. Here, to accommodate this challenge, we propose an interactive visualization system that helps HPC researchers understand the resiliency of HPC applications and compare their error propagation. Our system models an application's error propagation to study a program's resiliency by constructing and visualizing its fault tolerance boundary. Coordinating with multiple interactive designs, our system enables domain experts to efficiently explore the complicated spatial and temporal correlation between error propagations. At the end, the system integrated a nonmonotonic error propagation analysis with an adjustable graph propagation visualization to help domain experts examine the details of error propagation and answer such questions as why an error is mitigated or amplified by program execution.

97 MATHEMATICS AND COMPUTING↗

Leveraging Qubit Loss Detection in Fault-Tolerant Quantum Algorithms

Qubit loss errors constitute a dominant source of noise in many quantum hardware systems, particularly in neutral-atom quantum computers. We develop a theoretical framework to effectively detect and correct loss errors in logical algorithms and leverage such loss information in decoding. Considering general quantum error correction codes and logical circuits, we introduce a delayed-erasure decoder for experimentally motivated error models which leverages information from delayed loss detection to accurately correct loss errors, even when the precise moment of the error is unknown. Using this decoder, we identify strategies for detecting and correcting loss errors based on the logical circuit structure. For deep circuits prior to logical measurement, we explore methods to integrate loss detection into syndrome extraction with minimal overhead, identifying optimal strategies depending on the qubit loss fraction in the noise and hardware capabilities. In contrast, we find that many key algorithmic subroutines involve frequent gate teleportation, shortening the circuit depth before logical measurement and naturally replacing qubits with no additional experimental overhead. We simulate this setting using a toy model algorithm for small-angle synthesis and find a significant performance improvement as the loss fraction increases. These results provide a path forward for advancing large-scale fault-tolerant quantum computation in systems with loss error detection.

atoms↗

Towards Precision-Aware Fault Tolerance Approaches for Mixed-Precision Applications

Graphics Processing Units (GPUs), the dominantly adopted accelerators in HPC systems, are susceptible to transient hardware fault. New generation of GPUs feature mixed-precision architectures such as NVIDIA Tensor Cores to accelerate matrix multiplications. While being widely adapted, how would they behave under transient hardware faults remain unclear. In this study, we conduct a large-scale fault injection experiments on GEMM kernels implemented with different floating-point data types on the V100 and A100 Tensor Cores, and show distinct error resilience characteristics for the GEMMS with different formats. In the future, we plan to explore this space by building precision-aware floating-point fault tolerance techniques for applications such as DNNs that exercise low-precision computations.

Fang, Bo↗

Fault-tolerant grid frequency measurement algorithm during transients

Many critical electric grid operations rely on accurate grid frequency measurements. Unfortunately, the measurement accuracy can be easily undermined by power system transient faults. During a power system transient fault, the power grid voltages and currents are usually highly distorted by high-frequency components. What is worse, the power grid signals could have discontinuity during some system transient faults such as phase angle jump, and the discontinuity could result in large measurement errors to state-of-the-art grid measurement algorithms. In this study, a fault-tolerant grid frequency measurement algorithm during transients is proposed. The new algorithm consists of two stages. The first stage is a transient detector, and it can detect the occurrence of system transient faults instantaneously. The second stage is the intelligent frequency estimator, and it will adapt its measurements according to the transient detector. The performance of the algorithm is evaluated under different steady-state and transient conditions. Both dependability and security of the fault-tolerant algorithm are assessed by using PSCAD simulation data and IEEE Standard test data.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Evidence for the utility of quantum computing before fault tolerance

Quantum computing promises to offer substantial speed-ups over its classical counterpart for certain problems. However, the greatest impediment to realizing its full potential is noise that is inherent to these systems. The widely accepted solution to this challenge is the implementation of fault-tolerant quantum circuits, which is out of reach for current processors. Here we report experiments on a noisy 127-qubit processor and demonstrate the measurement of accurate expectation values for circuit volumes at a scale beyond brute-force classical computation. We argue that this represents evidence for the utility of quantum computing in a pre-fault-tolerant era. These experimental results are enabled by advances in the coherence and calibration of a superconducting processor at this scale and the ability to characterize and controllably manipulate noise across such a large device. We establish the accuracy of the measured expectation values by comparing them with the output of exactly verifiable circuits. In the regime of strong entanglement, the quantum computer provides correct results for which leading classical approximations such as pure-state-based 1D (matrix product states, MPS) and 2D (isometric tensor network states, isoTNS) tensor network methods break down. These experiments demonstrate a foundational tool for the realization of near-term quantum applications.

97 MATHEMATICS AND COMPUTING↗

Peer-to-Peer Communication Trade-Offs for Smart Grid Applications: Preprint

Peer-to-peer energy management systems for smart grids require developers to consider the trade-offs between the amount of communication traffic generated and the quality and speed of convergence of the control algorithms that are deployed. Employing a fully connected communication causes messages to scale exponentially with the number of nodes, while using a sparse connectivity causes less information dissemination leading to degradation of the algorithm performance. The best communication topology for a particular application lies somewhere in between and often requires empirical evaluation by application designers. Existing methods do not put focus on the needs for smart grid applications, which is information dissemination throughout the network and they do not provide a flexible solution for application developers to prototype and deploy different topologies without modifying the application code. This paper introduces a configurable virtual communication topology framework TopLinkMgr, allowing users to specify any chosen communication topology and deploy peer-to-peer applications using it. It also introduces a self-adaptive, fault-tolerant topology management algorithm, Bounded Path Dissemination that can ensure the dissemination of information to all peers within a specified threshold for a sparsely connected topology. Experiments show that the algorithm improves on convergence speed and accuracy over state-of-the-art methods and is also robust against node failures. The results indicate the possibility of achieving a close-to optimal convergence without overloading the network allowing the realization of peer-to-peer control platforms covering larger and more complex power systems.

Bounded Path Dissemination↗