Search NASA⌕ Search

SEARCH · Search NASA

Results for “Fault Tolerant Systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2013

Award ER25750: Coordinated Infrastructure for Fault Tolerance Systems Indiana University Final Report

The main purpose of the Coordinated Infrastructure for Fault Tolerance in Systems initiative has been to conduct research with a goal of providing end-to-end fault tolerance on a systemwide basis for applications and other system software. While fault tolerance has been an integral part of most high-performance computing (HPC) system software developed over the past decade, it has been treated mostly as a collection of isolated stovepipes. Visibility and response to faults has typically been limited to the particular hardware and software subsystems in which they are initially observed. Little fault information is shared across subsystems, allowing little flexibility or control on a system-wide basis, making it practically impossible to provide cohesive end-to-end fault tolerance in support of scientific applications. As an example, consider faults such as communication link failures that can be seen by a network library but are not directly visible to the job scheduler, or consider faults related to node failures that can be detected by system monitoring software but are not inherently visible to the resource manager. If information about such faults could be shared by the network libraries or monitoring software, then other system software, such as a resource manager or job scheduler, could ensure that failed nodes or failed network links were excluded from further job allocations and that further diagnosis could be performed. As a founding member and one of the lead developers of the Open MPI project, our efforts over the course of this project have been focused on making Open MPI more robust to failures by supporting various fault tolerance techniques, and using fault information exchange and coordination between MPI and the HPC system software stack from the application, numeric libraries, and programming language runtime to other common system components such as jobs schedulers, resource managers, and monitoring tools.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Redundancy Does Not Imply Fault Tolerance: Analysis of Distributed Storage Reactions to File-System Faults

In this paper, we analyze how modern distributed storage systems behave in the presence of file-system faults such as data corruption and read and write errors. We characterize eight popular distributed storage systems and uncover numerous problems related to file-system fault tolerance. We find that modern distributed systems do not consistently use redundancy to recover from file-system faults: a single file-system fault can cause catastrophic outcomes such as data loss, corruption, and unavailability. We also find that the above outcomes arise due to fundamental problems in file-system fault handling that are common across many systems. Finally, our results have implications for the design of next-generation fault-tolerant distributed and cloud storage systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Blockchain-based fault tolerant grid operations system

To maintain/improve the distribution system fault-tolerance for a grid with high penetration of distributed energy resources requires improvement, Blockchain can add value to improve fault-tolerant grid operations. That can be achieved using blockchain's core features of distributed consensus-based decision-making process and immutability.

Bereta dos Reis, Fernando↗
DOE OSTI2021

Blockchain for Fault-Tolerant Grid Operations

Distribution system fault/failure has a direct impact on customers. Distribution systems are not designed with the contingency of one or more elements failing as the transmission system, however, distribution systems are more likely to have faults in comparison to transmission and generation systems. Most of the distribution systems have a radial design with the protection system assumption of unidirectional power flow. The radial design results in the disconnection of customers for any given component failure. The rapid deployment of distributed energy resources conflicts with the assumption of unidirectional power flow. To maintain/improve the distribution system fault tolerance for a grid with high penetration of distributed energy resources requires improvement. Blockchain can add value to improve fault-tolerant grid operations. That can be achieved using blockchain’s core features of distributed consensus-based decision-making process and immutability. In the process of preparing for an event or system restoration, there is a need for reliable data and system situational awareness. The system situational awareness enables accurate planning for possible scenarios and system restoration after an event, an area where blockchain’s distributed architecture can help. The value of blockchain for a fault-tolerant grid operation is elaborated with three use cases: 1) fault location, isolation, and service restoration; 2) topology identification; and 3) data configuration. These use cases demonstrate how blockchain-based architecture can facilitate bridging the vulnerabilities of the existing implementation.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2017

What does fault tolerant Deep Learning need from MPI?

Deep Learning (DL) algorithms have become the {\em de facto} Machine Learning (ML) algorithm for large scale data analysis. DL algorithms are computationally expensive -- even distributed DL implementations which use MPI require days of training (model learning) time on commonly studied datasets. Long running DL applications become susceptible to faults -- requiring development of a fault tolerant system infrastructure, in addition to fault tolerant DL algorithms. This raises an important question: {\em What is needed from MPI for designing fault tolerant DL implementations?} In this paper, we address this problem for permanent faults. We motivate the need for a fault tolerant MPI specification by an in-depth consideration of recent innovations in DL algorithms and their properties, which drive the need for specific fault tolerance features. We present an in-depth discussion on the suitability of different parallelism types (model, data and hybrid); a need (or lack thereof) for check-pointing of any critical data structures; and most importantly, consideration for several fault tolerance proposals (user-level fault mitigation (ULFM), Reinit) in MPI and their applicability to fault tolerant DL implementations. We leverage a distributed memory implementation of Caffe, currently available under the Machine Learning Toolkit for Extreme Scale (MaTEx). We implement our approaches by extending MaTEx-Caffe for using ULFM-based implementation. Our evaluation using the ImageNet dataset and AlexNet neural network topology demonstrates the effectiveness of the proposed fault tolerant DL implementation using OpenMPI based ULFM.

Amatya, Vinay C.↗
DOE OSTI2016

Thermoelectric-Driven Sustainable Sensing and Actuation Systems for Fault-Tolerant Nuclear Incidents

The Fukushima Daiichi nuclear incident in March 2011 represented an unprecedented stress test on the safety and backup systems of a nuclear power plant. The lack of reliable information from key components due to station blackout was a serious setback, leaving sensing, actuation, and reporting systems unable to communicate, and safety was compromised. Although there were several independent backup power sources for required safety function on site, ultimately the batteries were drained and the systems stopped working. If, however, key system components were instrumented with self-powered sensing and actuation packages that could report indefinitely on the status of the system, then critical system information could be obtained while providing core actuation and control during off-normal status for as long as needed. This research project focused on the development of such a self-powered sensing and actuation system. The electrical power is derived from intrinsic heat in the reactor components, which is both reliable and plentiful. The key concept was based around using thermoelectric generators that can be integrated directly onto key nuclear components, including pipes, pump housings, heat exchangers, reactor vessels, and shielding structures, as well as secondary-side components. Thermoelectric generators are solid-state devices capable of converting heat directly into electricity. They are commercially available technology. They are compact, have no moving parts, are silent, and have excellent reliability. The key components to the sensor package include a thermoelectric generator (TEG), microcontroller, signal processing, and a wireless radio package, environmental hardening to survive radiation, flooding, vibration, mechanical shock (explosions), corrosion, and excessive temperature. The energy harvested from the intrinsic heat of reactor components can be then made available to power sensors, provide bi-directional communication, recharge batteries for other safety systems, etc. Such an approach is intrinsically fault tolerant: in the event that system temperatures increase, the amount of available energy will increase, which will make more power available for applications. The system can also be used during normal conditions to provide enhanced monitoring of key system components.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2022

Experimental demonstration of continuous quantum error correction

The storage and processing of quantum information are susceptible to external noise, resulting in computational errors. A powerful method to suppress these effects is quantum error correction. Typically, quantum error correction is executed in discrete rounds, using entangling gates and projective measurement on ancillary qubits to complete each round of error correction. Here we use direct parity measurements to implement a continuous quantum bit-flip correction code in a resource-efficient manner, eliminating entangling gates, ancillary qubits, and their associated errors. An FPGA controller actively corrects errors as they are detected, achieving an average bit-flip detection efficiency of up to 91%. Furthermore, the protocol increases the relaxation time of the protected logical qubit by a factor of 2.7 over the relaxation times of the bare comprising qubits. Our results showcase resource-efficient stabilizer measurements in a multi-qubit architecture and demonstrate how continuous error correction codes can address challenges in realizing a fault-tolerant system.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2024

AutoCheck: Automatically Identifying Variables for Checkpointing by Data Dependency Analysis

Checkpoint/Restart (C/R) has been widely deployed in numerous HPC systems, Clouds, and industrial data centers, which are typically operated by system engineers. Nevertheless, there is no existing approach that helps system engineers without domain expertise and domain scientists without system fault tolerance knowledge identify those critical variables accounted for correct application execution restoration in a failure for C/R. To address this problem, we propose an analytical model and a tool (AutoCheck) that can automatically identify critical variables to checkpoint for C/R. AutoCheck relies on first, analytically tracking and optimizing data dependency between variables and other application execution state, and second, a set of heuristics that identify critical variables for checkpointing from the refined data dependency graph (DDG). AutoCheck allows programmers to pinpoint critical variables to checkpoint quickly within a few minutes. We evaluate AutoCheck on 13 representative HPC benchmarks, demonstrating that AutoCheck can efficiently identify correct critical variables to checkpoint.

HPC↗
DOE OSTI2023

Fault isolation and fault-tolerant control for nonlinear stochastic distribution control systems with multiplicative faults

Here, in this paper, a fault isolation, diagnosis and fault tolerant control algorithm is proposed for nonlinear multiple multiplicative faults stochastic distribution control systems employing Takagi–Sugeno fuzzy system. To obtain the detailed fault information, a fault detection algorithm is introduced to discover the fault occurrence time. Then a fault isolation observer is built to produce the residual, and the error system is separated to subsystems affected only by disturbance and multiplicative faults. Moreover, a fault estimation scheme is presented to obtain the fault magnitude information. When faults occur, the system output probability density function will deviate from the desired distribution. So the model predictive control fault tolerant control scheme is needed to minimize the impact of faults as much as possible to make sure that the post fault output probability density function track the desired probability density function. The validity of the designed algorithm is demonstrated through a simulation example, where the fault tolerant control algorithm ensures that the system output probability density function still track the given output probability density function despite the complex case of multiple multiplicative faults occurring simultaneously.

42 ENGINEERING↗
DOE OSTI2021

Model Predictive Fault-Tolerant Tracking Control for PDF Control Systems With Packet Losses

In this article, a fault-tolerant tracking control strategy is investigated for nonlinear probability density function (PDF) control systems with the actuator fault, uncertainties, unknown disturbance, and random packet losses. The control input signal dropout and measurement signal dropouts are described as the independent Bernoulli distribution. An adaptive fault diagnosis (FD) observer based on the Lyapunov function is given to simultaneously estimate the fault, disturbance, and state with packet losses. Furthermore, different from the traditional robust fault-tolerant control (FTC), a new active fault-tolerant tracking controller is designed based on the model predictive control framework, which has better adaptive fault-tolerant performance. Finally, the validity of the proposed FTC method has been proved by a simulation study of a papermaking process.

42 ENGINEERING↗
DOE OSTI2015

Methods and apparatuses for self-generating fault-tolerant keys in spread-spectrum systems

Self-generating fault-tolerant keys for use in spread-spectrum systems are disclosed. At a communication device, beacon signals are received from another communication device and impulse responses are determined from the beacon signals. The impulse responses are circularly shifted to place a largest sample at a predefined position. The impulse responses are converted to a set of frequency responses in a frequency domain. The frequency responses are shuffled with a predetermined shuffle scheme to develop a set of shuffled frequency responses. A set of phase differences is determined as a difference between an angle of the frequency response and an angle of the shuffled frequency response at each element of the corresponding sets. Each phase difference is quantized to develop a set of secret-key quantized phases and a set of spreading codes is developed wherein each spreading code includes a corresponding phase of the set of secret-key quantized phases.

42 ENGINEERING↗
DOE OSTI2018

Collaborative operational fault tolerant control for stochastic distribution control system

Based on a class of industry processes, a distributed fault diagnosis and a collaborative operational fault tolerant control algorithms are presented for an irreversible interconnected stochastic distribution control system with boundary conditions. Different from the existed collaborative fault tolerant controllers, the controller does not need to change after fault occurs, the system tracking performance can be maintained by adjusting the setpoint of the whole system. Simulations are also included to show the effectiveness of the proposed algorithm and encouraging results have been obtained.

Stochastic distribution control, Fault diagnosis, ↗
DOE OSTI2025

Limitations of Fault-Tolerant Quantum Linear System Solvers for Quantum Power Flow

Quantum computers hold promise for solving problems intractable for classical computers, especially those with high time or space complexity. Practical quantum advantage can be said to exist for such problems when the end-to-end time for solving such a problem using a classical algorithm exceeds that required by a quantum algorithm. Reducing the power flow (PF) problem into a linear system of equations allows for the formulation of quantum PF (QPF) algorithms, which are based on solving methods for quantum linear systems such as the Harrow-Hassidim-Lloyd (HHL) algorithm. Speedup from using QPF algorithms is often claimed to be exponential when compared to classical PF solved by state-of-the-art algorithms. Here, we investigate the potential for practical quantum advantage in solving QPF compared to classical methods on gate-based quantum computers. Notably, this paper does not present a new QPF solving algorithm but scrutinizes the end-to-end complexity of the QPF approach, providing a nuanced evaluation of the purported quantum speedup in this problem. Our analysis establishes a best-case bound for the HHL-based quantum power flow complexity, conclusively demonstrating that the HHL-based method has higher runtime complexity compared to the classical algorithm for solving the direct current power flow (DCPF) and fast decoupled load flow (FDLF) problem. Notably, our analysis and conclusions can be extended to any quantum linear system solver with rigorous performance guarantees, based on the known complexity lower bounds for this problem. Additionally, we establish that for potential practical quantum advantage (PQA) to exist it is necessary to consider DCPF-type problems with a very narrow range of condition number values and readout requirements.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2013

Reverse Computation for Rollback-based Fault Tolerance in Large Parallel Systems

Reverse computation is presented here as an important future direction in addressing the challenge of fault tolerant execution on very large cluster platforms for parallel computing. As the scale of parallel jobs increases, traditional checkpointing approaches suffer scalability problems ranging from computational slowdowns to high congestion at the persistent stores for checkpoints. Reverse computation can overcome such problems and is also better suited for parallel computing on newer architectures with smaller, cheaper or energy-efficient memories and file systems. Initial evidence for the feasibility of reverse computation in large systems is presented with detailed performance data from a particle simulation scaling to 65,536 processor cores and 950 accelerators (GPUs). Reverse computation is observed to deliver very large gains relative to checkpointing schemes when nodes rely on their host processors/memory to tolerate faults at their accelerators. A comparison between reverse computation and checkpointing with measurements such as cache miss ratios, TLB misses and memory usage indicates that reverse computation is hard to ignore as a future alternative to be pursued in emerging architectures.

Perumalla, Kalyan S↗
DOE OSTI2019

FTLADS: Object-Logging Based Fault-Tolerant Big Data Transfer System Using Layout Aware Data Scheduling

The layout-aware data scheduling (LADS) data movement framework optimizes congestion for end-to-end data transfers. During data transfer, LADS can avoid congested storage elements by exploiting the underlying storage layout at each endpoint. This improves the I/O bandwidth and hence the data transfer rate across high-speed networks. However, the absence of fault tolerance (FT) in LADS results in data retransmission overhead and may lead to possible data integrity issues upon faults. In this paper, we propose object-logging FT mechanisms to avoid transmitting the objects that are successfully written into the parallel file system (PFS) at the sink end. Depending on the number of log files created for the whole dataset, we have classified our FT mechanisms into three different categories: file logger, transaction logger, and universal logger. Also, to address the space overhead, we have proposed different methods of populating the log files with the information of the successfully transferred objects. We have evaluated the data transfer performance and recovery time overhead of the proposed object-logging-based FT mechanisms on the LADS data transfer framework. Our experimental results reflect that FT mechanisms exhibit negligible overhead (<; 1%) with respect to the data transfer time. Yet, the fault recovery time is 10% higher than the total data transfer time at any fault point.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

A Hot‐Swappable, Fault‐Tolerant, Modular Power Converter System for Solar Photovoltaic Plants

The performance metrics of the state-of-the-art commercial solar inverters, such as system cost, operation and maintenance (O&M) cost, service life, reliability, maintainability, and power density are much lower than the target metrics needed to achieve SunShot’s 2030 levelized cost of energy (LCOE) goals. To overcome the shortcomings of the existing solar inverters, this project proposed a novel Hot-Swappable, Fault-Tolerant, Modular Power Converter (HSFT-MPC) concept for solar photovoltaic (PV) plants and proved the concept through the design, fabrication, and laboratory test validation of a single-phase HSFT-MPC prototype. The HSFT-MPC has the following distinct advantages over the state-of-the-art: 1) elimination of harmonic/ electromagnetic interference (EMI) filter in the inverter stage due to the novel topology, 2) lower system cost and higher power density due to the modular design, elimination of harmonic/EMI filter, and lower cooling requirement, 3) higher efficiency due to lower switching frequencies, 4) higher reliability and longer (50 years) service life due to simpler cooling and fault tolerance capability, 5) easier installation, lower O&M cost, and improved maintainability due to the modular design and hot-swappable power electronic building blocks (PEBBs), and 6) improved manufacturability due to the modular design. This project developed a single-phase HSFT-MPC prototype with 25kW nominal output power, 2.4kV, 60Hz nominal AC output, lower than 5% AC output voltage total harmonic distortion, over 5 kW/L inverter power density, and 99.4% inverter peak efficiency, being tolerant to failure of single and multiple PEBBs, and capable of hot swapping of the failed PEBB(s). The HSFT-MPC enables uninterruptable operation of the solar PV plant when failure of single or multiple PEBBs or PV modules occurs. Compared with the existing solar inverters in the market, the HSFT-MPC is expected to reduce the inverter failure-caused downtime and energy losses of solar PV plants by more than 60% and 50%, respectively. Project findings have been presented at major conferences in the field and published in peer-reviewed papers, which added new knowledge to the field of power electronics for solar PV systems. A minicourse on Solar PV Systems was developed for outreach activities. The minicourse will help attract young individuals to the renewable energy profession which has a significant talent shortage. The HSFT-MPC is expected to overcome all of the shortcomings of the state-of-the-art solar inverters in terms of cost, efficiency, service life, reliability, maintainability, and manufacturability targets needed to achieve SunShot’s 2030 LCOE goals. Therefore, the proposed HSFT-MPC concept has the great potential to disrupt the current solar inverter market. This project created a pathway towards industry adoption of the HSFT-MPC to help achieve 50-year service life solar PV systems. Since the solar PV plants using the HSFT-MPC will feature with higher reliability, longer service life, and easier maintenance, they are particularly useful for the rural areas with underserved populations that demand reliable and affordable clean electricity. The outcomes of the project have the strong potential to address national needs in the field of renewable energy to reduce CO 2 emissions from the electricity sector, reduce imports of energy from foreign sources, and improve energy security, efficiency, and sustainability. Since electricity is used in almost all of society’s sectors, the outcomes of the project will benefit various sectors of society and economy.

14 SOLAR ENERGY↗