Search NASA⌕ Search

SEARCH · Search NASA

Results for “Fault Tolerant Algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Preserving Collective Performance Across Process Failure for a Fault Tolerant MPI

Application developers are investigating Algorithm Based Fault Tolerance (ABFT) techniques to improve the efficiency of application recovery beyond what traditional techniques alone can provide. Applications will depend on libraries to sustain failure-free performance across process failure to continue to efficiently use High Performance Computing (HPC) systems even in the presence of process failure. Optimized Message Passing Interface (MPI) collective operations are a critical component of many scalable HPC applications. However, most of the collective algorithms are not able to handle process failure. Next generation MPI implementations must provide fault aware versions of such algorithms that can sustain performance across process failure. This paper discusses the design and implementation of fault aware collective algorithms for tree structured communication patterns. The three design approaches of rerouting, lookup avoiding and rebalancing are described, and analyzed for their performance impact relative to a similar fault unaware collective algorithm. The analysis shows that the rerouting approach causes up to a four times performance degradation while the rebalancing approach can bring the performance within 1% of the fault unaware performance. Additionally, this paper introduces the reader to a set of run-through stabilization semantics being developed by the MPI Forum's Fault Tolerance Working Group to support ABFT. This paper underscores the need for care to be taken when designing new fault aware collective algorithms for fault tolerant MPI implementations.

99 GENERAL AND MISCELLANEOUS↗

Flight test results of failure detection and isolation algorithms for a redundant strapdown inertial measurement unit

Flight test results for two sensor fault-tolerant algorithms developed for a redundant strapdown inertial measurement unit are presented. The inertial measurement unit (IMU) consists of four two-degrees-of-freedom gyros and accelerometers mounted on the faces of a semi-octahedron. Fault tolerance is provided by edge vector test and generalized likelihood test algorithms, each of which can provide dual fail-operational capability for the IMU. To detect the wide range of failure magnitudes in inertial sensors, which provide flight crucial information for flight control and navigation, failure detection and isolation are developed in terms of a multi level structure. Threshold compensation techniques, developed to enhance the sensitivity of the failure detection process to navigation level failures, are presented. Four flight tests were conducted in a commercial transport-type environment to compare and determine the performance of the failure detection and isolation methods. Dual flight processors enabled concurrent tests for the algorithms. Failure signals such as hard-over, null, or bias shift, were added to the sensor outputs as simple or multiple failures during the flights. Both algorithms provided timely detection and isolation of flight control level failures. The generalized likelihood test algorithm provided more timely detection of low-level sensor failures, but it produced one false isolation. Both algorithms demonstrated the capability to provide dual fail-operational performance for the skewed array of inertial sensors.

Morrell, F. R.↗

Self-stabilizing Connected Components

For the problem of computing the connected components of a graph, this paper considers the design of algorithms that are resilient to transient hardware faults, like bit Hips. More specifically, it applies the technique of self-stabilization. A system is self-stabilizing if, when starting from a valid or invalid state, it is guaranteed to reach a valid state after a finite number of steps. Therefore on a machine subject to a transient fault, a self-stabilizing algorithm could recover if that fault caused the system to enter an invalid state. We give a comprehensive analysis of the valid and invalid states during label propagation and derive algorithms to verify and correct the invalid state. The self-stabilizing label-propagation algorithm performs O (V log V) additional computation and requires O (V) additional storage over its conventional counterpart (and, as such, does not increase asymptotic complexity over conventional label propagation). When run against a battery of simulated fault injection tests, the self-stabilizing label propagation algorithm exhibits more resilient behavior than a triple modular redundancy (TMR) based fault-tolerant algorithm in 80% of cases. From a performance perspective, it also outperforms TMR as it requires fewer iterations in total. Beyond the fault tolerance properties of self-stabilizing label-propagation, we believe, they are useful from the theoretical perspective; and may have other use-cases.

Sao, Piyush↗

Copilot: Monitoring Embedded Systems

Runtime verification (RV) is a natural fit for ultra-critical systems, where correctness is imperative. In ultra-critical systems, even if the software is fault-free, because of the inherent unreliability of commodity hardware and the adversity of operational environments, processing units (and their hosted software) are replicated, and fault-tolerant algorithms are used to compare the outputs. We investigate both software monitoring in distributed fault-tolerant systems, as well as implementing fault-tolerance mechanisms using RV techniques. We describe the Copilot language and compiler, specifically designed for generating monitors for distributed, hard real-time systems. We also describe two case-studies in which we generated Copilot monitors in avionics systems.

Pike, Lee↗

Risk assessment of safety-critical data communication in digital safety feature control system - 205

As one of the safety-critical systems in Advanced Power Reactor-1400 (APR-1400) nuclear power plant (NPP), the Engineered Safety Feature-Component Control System (ESF-CCS) employs safety data link and network communication for transmitting safety component actuation to effectively facilitate various safety-critical field controllers. However, data communication failure risk in the ESF-CCS has yet to be fully quantified, the probability that a safety-critical system in digitalized NPP becomes unsafe due to a network failure must be evaluated to quantify the risk of digital I and C system. In this study, a fault tree model was developed to assess the data link and data network failure-induced unavailability of a safety-critical system function used to generate an automated control signal for actuating design basis accident mitigation equipment. Especially, the risk of data communication failure in a digital safety feature control system was analyzed in consideration of interconnection between safety-critical controllers and the fault-tolerant algorithm implemented in the network configuration. Based on the developed fault tree model, case studies were performed to quantitatively assess the unavailability of ESF-CCS signal generation due to safety data link and network failure and their risk effect on safety signal generation unavailability. This study is expected to provide risk information on the safety-critical data communication in a digitalized NPP instrumentation and control system. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Flight demonstration of redundancy management algorithms for a skewed array of inertial sensors

Flight test results for two fault-tolerance algorithms developed for a redundant strapdown inertial measurement unit consisting of four 2-DOF gyros and accelerometers mounted on the faces of a semioctahedron are presented. Although both algorithms provided timely detection and isolation of flight control level failures, the generalized likelihood test algorithm provided more timely detection and isolation of low-level sensor failures than the edge vector test algorithm. The generalized likelihood test produced a false isolation for the case of a dual low-level failure applied to the sensitive axes of an accelerometer. Both of the algorithms were shown to provide dual fail-operational performance for the skewed array of inertial sensors.

Morrell, F. R.↗

Resilient Autonomous Wind Farms: Preprint

With the advent of an increasing number of control strategies that seek to optimize wind turbine performance on a farm-level, taking account of individual wind turbine information to achieve wind farm-level objectives has become an increasingly important goal. Methods for controlling wind turbines on an individual and farm level have seen significant development, and an abundance of new implementations for gathering and using data from turbines have created potential for novel control mechanisms which can further optimize the performance and delivery characteristics of a wind farm. A key element of making these wind farms more efficient is to develop reliable algorithms that use local sensor information that is already being collected, such as supervisory control and data acquisition (SCADA) data, local meteorological stations, and nearby radars/sodars/lidars. Making use of information from all wind turbines in a wind farm can enable such approaches as determining the atmospheric conditions across the farm, improving fault-finding, and enabling more efficient overall control of farm-wide optimizations through mechanisms such as wake-steering. However, these approaches typically involve a centralized communications and control center. In order to ensure the resilient operation of the farm, it is necessary to develop an approach which distributes the calculation and communication amongst multiple nodes throughout the farm. In this fashion, a redundant, robust, and secure network can be created, which can tolerate faults in calculation, communication, and even external attacks which seek to disrupt the operation of the wind farm. This paper introduces the use of the Raft Byzantine Fault Tolerance algorithm in the implementation of autonomous control of a wind farm. This implementation will allow for fault tolerance for malfunctioning nodes, sensors, transmitters, and connectors. This approach is equally extensible to account for malicious actors. It will be shown to achieve overall consensus, provided the number of faults/malicious nodes is less than 3$n$+1, where $n$ is the number of turbine cluster faults which may occur, and to be robust in the face of multiple arbitrary faults.

autonomous↗

Resilient Autonomous Wind Farms

With the advent of an increasing number of control strategies that seek to optimize wind turbine performance on a farm level, taking into account individual wind turbine information to achieve wind-farm-level objectives has become an increasingly important goal. Methods for controlling wind turbines on an individual and farm level have experienced significant development, and an abundance of new implementations for gathering and using data from turbines have created potential for novel control mechanisms that can further optimize the performance and delivery characteristics of a wind farm. A key element of making these wind farms more efficient is to develop reliable algorithms that use local sensor information that is already being collected, such as from local meteorological stations, nearby radars, sodars, and lidars, and supervisory control and data acquisition (SCADA) data. Making use of information from all wind turbines in a wind farm can enable such approaches as determining the atmospheric conditions across the farm, improving fault-finding, and ensuring more efficient overall control of farmwide optimizations through mechanisms such as wake steering. However, these approaches typically involve a centralized communications and control center. In order to ensure the resilient operation of the farm, it is necessary to develop an approach that distributes the calculation and communication amongst multiple nodes throughout the farm. In this fashion, a redundant, robust, and secure network can be created, which can tolerate faults in calculation, communication, and even external attacks that seek to disrupt the operation of the wind farm. This paper introduces the use of the Raft-Byzantine-Fault-Tolerant algorithm in the implementation of autonomous control of a wind farm. This implementation will allow for fault tolerance for malfunctioning nodes, sensors, transmitters, and connectors. This approach is equally extensible to account for malicious actors. It will...

fault tolerance↗

Flight test results of a vector-based failure detection and isolation algorithm for a redundant strapdown inertial measurement unit

Flight test results of a vector-based fault-tolerant algorithm for a redundant strapdown inertial measurement unit are presented. Because the inertial sensors provide flight-critical information for flight control and navigation, failure detection and isolation is developed in terms of a multi-level structure. Threshold compensation techniques for gyros and accelerometers, developed to enhance the sensitivity of the failure detection process to low-level failures, are presented. Four flight tests, conducted in a commercial transport type environment, were used to determine the ability of the failure detection and isolation algorithm to detect failure signals, such a hard-over, null, or bias shifts. The algorithm provided timely detection and correct isolation of flight control- and low-level failures. The flight tests of the vector-based algorithm demonstrated its capability to provide false alarm free dual fail-operational performance for the skewed array of inertial sensors.

Morrell, F. R.↗

Fault-tolerant grid frequency measurement algorithm during transients

Many critical electric grid operations rely on accurate grid frequency measurements. Unfortunately, the measurement accuracy can be easily undermined by power system transient faults. During a power system transient fault, the power grid voltages and currents are usually highly distorted by high-frequency components. What is worse, the power grid signals could have discontinuity during some system transient faults such as phase angle jump, and the discontinuity could result in large measurement errors to state-of-the-art grid measurement algorithms. In this study, a fault-tolerant grid frequency measurement algorithm during transients is proposed. The new algorithm consists of two stages. The first stage is a transient detector, and it can detect the occurrence of system transient faults instantaneously. The second stage is the intelligent frequency estimator, and it will adapt its measurements according to the transient detector. The performance of the algorithm is evaluated under different steady-state and transient conditions. Both dependability and security of the fault-tolerant algorithm are assessed by using PSCAD simulation data and IEEE Standard test data.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Low-overhead transversal fault tolerance for universal quantum computation

Fast, reliable logical operations are essential for realizing useful quantum computers. By redundantly encoding logical qubits into many physical qubits and using syndrome measurements to detect and correct errors, we can achieve low logical error rates. However, for many practical quantum error correction codes such as the surface code, owing to syndrome measurement errors, standard constructions require multiple extraction rounds—of the order of the code distance d—for fault-tolerant computation, particularly considering fault-tolerant state preparation. Here we show that logical operations can be performed fault-tolerantly with only a constant number of extraction rounds for a broad class of quantum error correction codes, including the surface code with magic state inputs and feedforward, to achieve ‘transversal algorithmic fault tolerance’. Through the combination of transversal operations7 and new strategies for correlated decoding, despite only having access to partial syndrome information, we prove that the deviation from the ideal logical measurement distribution can be made exponentially small in the distance, even if the instantaneous quantum state cannot be made close to a logical codeword because of measurement errors. We supplement this proof with circuit-level simulations in a range of relevant settings, demonstrating the fault tolerance and competitive performance of our approach. Furthermore, our work sheds new light on the theory of quantum fault tolerance and has the potential to reduce the space–time cost of practical fault-tolerant quantum computation by over an order of magnitude.

Zhou, Hengyun [QuEra Computing, Boston, MA (United↗

A case for Virtual Machine based Fault Injection in a High-Performance Computing Environment

Large-scale computing platforms provide tremendous capabilities for scientific discovery. These systems have hundreds of thousands of computing cores, hundreds of terabytes of memory, and enormous high-performance interconnection networks. These systems are facing enormous challenges to achieve performance at such scale. Failures are an Achilles heel of these enormous systems. As applications and system software scale up to multi-petaflop and beyond to exascale platforms, the occurrence of failure will be much more common. This has given rise to a push in fault-tolerance and resilience research for HPC systems. This includes work on log analysis to identify types of failures, enhancements to the Message Passing Interface (MPI) to incorporate fault awareness, and a variety of fault tolerance mechanisms that span redundant computation, algorithm based fault tolerance, and advanced checkpoint/restart techniques. While there is much work to be done on the FT/Resilience mechanisms for such large-scale systems, there is also a profound gap in the tools for experimentation. This gap is compounded by the fact that HPC environments have stringent performance requirements and are often highly customized. The tool chain for these systems are often tailored for the platform and while the majority of systems on the Top500 Supercomputer list run Linux, these operating environments typically contain many site/machine specific enhancements. Therefore, it is desirable to maintain a consistent execution environment to minimize end-user (scientist) interruption. The work on system-level virtualization for HPC system offers a unique opportunity to maintain a consistent execution environment via a virtual machine (VM). Recent work on virtualization for HPC has shown that low-overhead, high performance systems can be realized. Virtualization also provides a clean abstraction for building experimental tools for investigation into the effects of failures in HPC and the related research on FT/Resilience mechanisms and policies. In this paper we discuss the motivation for tools to perform fault injection in an HPC context, and outline an approach that can leverage virtualization.

99 GENERAL AND MISCELLANEOUS↗

Adapting Secure MultiParty Computation to Support Machine Learning in Radio Frequency Sensor Networks

In this project we developed and validated algorithms for privacy-preserving linear regression using a new variant of Secure Multiparty Computation (MPC) we call "Hybrid MPC" (hMPC). Our variant is intended to support low-power, unreliable networks of sensors with low-communication, fault-tolerant algorithms. In hMPC we do not share training data, even via secret sharing. Thus, agents are responsible for protecting their own local data. Only the machine learning (ML) model is protected with information-theoretic security guarantees against honest-but-curious agents. There are three primary advantages to this approach: (1) after setup, hMPC supports a communication-efficient matrix multiplication primitive, (2) organizations prevented by policy or technology from sharing any of their data can participate as agents in hMPC, and (3) large numbers of low-power agents can participate in hMPC. We have also created an open-source software library named "Cicada" to support hMPC applications with fault-tolerance. The fault-tolerance is important in our applications because the agents are vulnerable to failure or capture. We have demonstrated this capability at Sandia's Autonomy New Mexico laboratory through a simple machine-learning exercise with Raspberry Pi devices capturing and classifying images while flying on four drones.

42 ENGINEERING↗

Analysis and design of algorithm-based fault-tolerant systems

An important consideration in the design of high performance multiprocessor systems is to ensure the correctness of the results computed in the presence of transient and intermittent failures. Concurrent error detection and correction have been applied to such systems in order to achieve reliability. Algorithm Based Fault Tolerance (ABFT) was suggested as a cost-effective concurrent error detection scheme. The research was motivated by the complexity involved in the analysis and design of ABFT systems. To that end, a matrix-based model was developed and, based on that, algorithms for both the design and analysis of ABFT systems are formulated. These algorithms are less complex than the existing ones. In order to reduce the complexity further, a hierarchical approach is developed for the analysis of large systems.

Nair, V. S. Sukumaran↗

What FM can offer DFCS design

The results of aircrafts and spacecrafts flight tests are reported. It is shown that the problems of Digital Flight Control Systems (DFCS) are the problems of systems whose complexity has exceeded the reach of the intellectual tools employed. It is also shown that intuition, experience, and techniques derived from mechanical and analog systems are insufficient for complex, integrated, digital systems. Formal Methods (FM) of computer science can offer DFCS systematic techniques for the construction of trustworthy software, including: techniques for the precise specification of requirements and the development of designs; systematic approaches to the design and structuring of distributed and concurrent systems; fault tolerance algorithms; and systematic methods of testing and analytic methods of verification.

Rushby, John↗

A Methodology for Evaluating Artifacts Produced by a Formal Verification Process

The goal of this study is to produce a methodology for evaluating the claims and arguments employed in, and the evidence produced by formal verification activities. To illustrate the process, we conduct a full assessment of a representative case study for the Enabling Technology Development and Demonstration (ETDD) program. We assess the model checking and satisfiabilty solving techniques as applied to a suite of abstract models of fault tolerant algorithms which were selected to be deployed in Orion, namely the TTEthernet startup services specified and verified in the Symbolic Analysis Laboratory (SAL) by TTTech. To this end, we introduce the Modeling and Verification Evaluation Score (MVES), a metric that is intended to estimate the amount of trust that can be placed on the evidence that is obtained. The results of the evaluation process and the MVES can then be used by non-experts and evaluators in assessing the credibility of the verification results.

Siminiceanu, Radu I.↗

Fault isolation and fault-tolerant control for nonlinear stochastic distribution control systems with multiplicative faults

Here, in this paper, a fault isolation, diagnosis and fault tolerant control algorithm is proposed for nonlinear multiple multiplicative faults stochastic distribution control systems employing Takagi–Sugeno fuzzy system. To obtain the detailed fault information, a fault detection algorithm is introduced to discover the fault occurrence time. Then a fault isolation observer is built to produce the residual, and the error system is separated to subsystems affected only by disturbance and multiplicative faults. Moreover, a fault estimation scheme is presented to obtain the fault magnitude information. When faults occur, the system output probability density function will deviate from the desired distribution. So the model predictive control fault tolerant control scheme is needed to minimize the impact of faults as much as possible to make sure that the post fault output probability density function track the desired probability density function. The validity of the designed algorithm is demonstrated through a simulation example, where the fault tolerant control algorithm ensures that the system output probability density function still track the given output probability density function despite the complex case of multiple multiplicative faults occurring simultaneously.

42 ENGINEERING↗

Scalable distributed consensus to support MPI fault tolerance.

As system sizes increase, the amount of time in which an application can run without experiencing a failure decreases. Exascale applications will need to address fault tolerance. In order to support algorithm-based fault tolerance, communication libraries will need to provide fault-tolerance features to the application. One important fault-tolerance operation is distributed consensus. This is used, for example, to collectively decide on a set of failed processes. This paper describes a scalable, distributed consensus algorithm that is used to support new MPI fault-tolerance features proposed by the MPI 3 Forum's fault-tolerance working group. The algorithm was implemented and evaluated on a 4,096-core Blue Gene/P. The implementation was able to perform a full-scale distributed consensus in 305 {mu}s and scaled logarithmically.

99 GENERAL AND MISCELLANEOUS↗