Search NASA⌕ Search

SEARCH · Search NASA

Results for “fault tolerant systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Orchestrating Fault Prediction with Live Migration and Checkpointing

Checkpoint/Restart (C/R) is widely used to provide fault tolerance on High-Performance Computing (HPC) systems. However, Parallel File System (PFS) overhead and failure uncertainty cause significant application overhead. This paper develops an adaptive multi-level C/R model that incorporates a failure prediction and analysis model, which orchestrates failure prediction, checkpointing, checkpoint frequency, and proactive live migration along with the additional benefit of Burst Buffers (BB). It effectively reduces the overheads due to failures, checkpointing, and recovery. Simulation results for the Summit supercomputer yield a reduction of ~20%-86% in application overhead due to BBs, orchestrated failure prediction, and migration. We also observe a ~29% decrease in checkpoint writes to BBs, which can increase the longevity of the BB storage devices.

Behera, Subhendu↗
DOE OSTI2020

Stochastic Fault Detection

This entry describes the state-of-the-art and future perspectives on stochastic fault detection, namely, stochastic fault detection and diagnosis (FDD). Both model-based and data-driven FDD methods for stochastic signals and systems have been included, where the use of hypothesis testing, Kalman filtering, system estimation, principal component analysis (PCA), and stochastic distribution control has been discussed for the construction of effective FDD algorithms. Indeed, stochastic FDD constitute an important and integrated part in developing fault-tolerant controls (FTC) for guaranteed safe operation of control systems, of which increased penetration of random factors is inevitable nowadays.

Wang, Aiping↗
DOE OSTI2023

Distributed Ledger Technology for Fault Tolerant Distribution Grid Operations

This paper explores the potential of distributed ledger technology (DLT) to improve fault-tolerant grid operations by leveraging its core features as an immutable, decentralized ledger, a distributed, consensus-based agreement process, and a distributed state-replication engine. Distribution power systems deliver electricity to millions of customers; however, they are susceptible to various threats that can result in customer interruptions. These include faults caused by adverse weather conditions, natural disasters, vegetation growth, equipment failure, and malicious attacks. To minimize the effects of these faults, fault-handling approaches rely on network knowledge to isolate affected areas and reconnect unaffected areas, reducing the number of affected customers while maintaining safety. Here, we present a trusted data-sharing architecture that enables independent, distributed actors to reconstruct the pre-fault system state by enabling distributed resources to make appropriate decisions with limited network/system information. Although the process requires some data sharing between switch-delimited areas, the approach limits the amount of private information shared, preserving customers' privacy and business-sensitive information. We include three use cases that form a foundation for third parties to develop functional solutions that can eventually be deployed in the field. The gross error detection method used within switch-delimited areas can identify sensor errors and accurately detect circuit breaker states. The evaluation of possible reconnection while preserving data ownership resulted in a voltage magnitude difference smaller than 0.001% from the OpenDSS power flow solution that has full system knowledge, which is below the expected power flow tolerance. The approach offers a promising opportunity for improving fault-tolerant distribution grid operations.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Network Slicing for Federated Learning in Operational Technology Environment

Industrial Control Systems (ICS) and Supervisory Control and Data Acquisition (SCADA) environments are essential to modern infrastructure, facing challenges in ensuring low-latency, high-throughput communication while mitigating cyber threats. This paper presents a framework integrating Federated Learning (FL) and network slicing with Quality of Service (QoS) to enable real-time monitoring without disrupting OT operations. Leveraging digital twin technology and Network Function Virtualization (NFV), the architecture supports predictive analytics and Industry 4.0 requirements. FL facilitates decentralized model training, preserving data privacy and scalability, though it introduces potential throughput constraints. Network slicing addresses this by creating dedicated virtualized segments optimized for performance and security. Advanced fault tolerance at the container and instance levels enhances system reliability. The proposed architecture ensures high throughput, low latency, and secure orchestration for real-time anomaly detection in OT networks. Performance evaluations validate its efficiency in throughput, deployment, and learning accuracy, providing a robust foundation for future ICS automation and data-driven decision-making.

Delgado, Brian G. Rodiles [University of Texas at ↗
DOE OSTI2023

Impact Report: Quantum Systems Accelerator

The Quantum Systems Accelerator (QSA) is a U.S. National Quantum Information Science Research Center established in August 2020 and funded by the Department of Energy (DOE) Office of Science. QSA is composed of 15 partner institutions— universities and national laboratories—bringing together pioneers of many of today’s unique quantum information science (QIS) and engineering capabilities. Led by Lawrence Berkeley National Laboratory (Berkeley Lab), with Sandia National Laboratories (Sandia Labs) as the lead partner, 250+ QSA researchers are catalyzing U.S. leadership in a fast-growing field that seeks solutions to the Nation’s and the world’s most pressing problems by harnessing the laws of quantum mechanics.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

5G integrated edge computing platform for efficient component monitoring in coal-fired power plants

This project developed a cutting-edge 5G-integrated edge computing framework to enhance operational efficiency and reliability in coal-fired power plants through real-time component monitoring and anomaly detection. The initiative focused on leveraging distributed machine learning, federated learning, and 5G-based dynamic network slicing to support scalable, fault-tolerant monitoring environments to meet the operational requirements in industrial control systems. With a Distributed Edge Computing Service (DECS) orchestration, this project enabled federated learning at edge for condition monitoring and introduced adaptive client selection strategies to minimize communication overhead. Scalable distributed training was achieved using the Horovod framework, thus enhancing performance across edge nodes. In the realm of 5G networking, the project designed and deployed reconfigurable, QoS-aware network slicing tailored for operational technology (OT) environments, integrating software-defined networks to bolster cyber-resilience and enabling dynamic slicing for federated learning workloads. A significant milestone was the development of a virtualized ICS environment with 5G core integration—which allowed elastic and fault tolerant distributed training on real-world datasets such as NASA Bearings, Hydraulic Systems, and TEP. To broaden the impact of the project, a TRL-3 virtualized ICS testbed for research and education was designed. This project engaged several graduate and undergraduate students to conduct research on the cutting-edge technology, and it resulted in one PhD dissertation, one MS thesis, and over 14 peer-reviewed publications. With the support of this project students also participated in national cybersecurity competitions to improve their professional development skills.

20 FOSSIL-FUELED POWER PLANTS↗
DOE OSTI2024

Integrating HPC, AI, and Workflows for Scientific Data Analysis: Report from Dagstuhl Seminar 23352

The Dagstuhl Seminar 23352, titled “Integrating HPC, AI, and Workflows for Scientific Data Analysis,” held from August 27 to September 1, 2023, was a significant event focusing on the synergy between High-Performance Computing (HPC), Artificial Intelligence (AI), and scientific workflow technologies. The seminar recognized that modern Big Data analysis in science rests on three pillars: workflow technologies for reproducibility and steering, AI and Machine Learning (ML) for versatile analysis, and HPC for handling large data sets. These elements, while crucial, have traditionally been researched separately, leading to gaps in their integration. The seminar aimed to bridge these gaps, acknowledging the challenges and opportunities at the intersection of these technologies. The event highlighted the complex interplay between HPC, workflows, and ML, noting how ML has increasingly been integrated into scientific workflows, thereby enhancing resource demands and bringing new requirements to HPC architectures, like support for GPUs and iterative computations. The seminar also addressed the challenges in adapting HPC for large-scale ML tasks, including in areas like deep learning, and the need for workflow systems to evolve to leverage ML in data analysis fully. Moreover, the seminar explored how ML could optimize scientific workflow systems and HPC operations, such as through improved scheduling and fault tolerance. A key focus was on identifying prestigious use cases of ML in HPC and understanding their unique, unmet requirements. The stochastic nature of ML and its impact on the reproducibility of data analysis on HPC systems was also a topic of discussion.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Understanding the use of message passing interface in exascale proxy applications

The Exascale Computing Project (ECP) focuses on the development of future exascale-capable applications. Most ECP applications use the message passing interface (MPI) as their parallel programming model with mini-apps serving as proxies. This paper explores the explicit usage of MPI in such ECP proxy applications. We empirically analyze 14 proxy applications from the ECP Proxy Apps Suite. We use the MPI profiling interface (PMPI) to collect MPI usage patterns in ECP proxy apps. Our analysis shows that a small subset of features from MPI is commonly used in the proxies of exascale-capable applications, even when they reference third-party libraries. Overall, this study is intended to provide a better understanding of the use of MPI in current exascale applications. The findings can help focus software investments made for exascale systems in the MPI middleware including optimization, fault-tolerance, tuning, and hardware-offload.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Efficient Client Selection in Federated Learning

Federated Learning (FL) enables decentralized machine learning while preserving data privacy. This paper proposes a novel client selection framework that integrates differential privacy and fault tolerance. The adaptive client selection adjusts the number of clients based on performance and system constraints, with noise added to protect privacy. Evaluated on the UNSW-NB15 and ROAD datasets for network anomaly detection, the method improves accuracy by 7% and reduces training time by 25 % compared to baselines. Fault tolerance enhances robustness with minimal performance trade-offs.

Marfo, William [University of Texas at El Paso,Dep↗
DOE OSTI2022

Oak Ridge National Laboratory Pilot Demonstration of an Attestation and Anomaly Detection Framework using Distributed Ledger Technology for Power Grid Infrastructure

This report summarizes the design and pilot demonstration of a framework called Grid Guard that was created to provide increased data and device trustworthiness to electric grid devices by leveraging distributed ledger technology (DLT), specifically blockchain. Grid Guard contains a combination of core cryptographic methods such as the secure hash algorithm (SHA), and asymmetric cryptography, private permissioned blockchain, baselining configuration data, consensus algorithm (Raft) and the Hyperledger Fabric (HLF) framework. The system implements a low energy, fast, and robust enhancement to system trustworthiness within and across electric grid systems such as substations, control centers and metering infrastructures. Blockchain is a distributed database structured that provides a practically unalterable (immutable) timeline of stored transactions. By relying on hashing and the Raft consensus algorithm, if an entity tries to illegitimately alter a record at one instance of the database the other ledger nodes are not altered. They work to cross-reference each other and easily locate any incorrectly added data and remove it. The bulk raw data is stored in an off-chain storage (outside of the blockchain ledger) and a hash of this baseline data is stored in the Blockchain ledger via hashing windows of time-series and configuration data, after aggregation and filtering. The bulk off-chain data repository is then considered to be trust-anchored using the hashes stored in the blockchain. To secure the electric grid testbed devices and data, device configuration baselines were compared to those baselines that had been previously stored in the ledger. Statistical baselines for device configurations, network communication patterns, and high-speed sensor data are calculated and then stored off-chain and hashes stored in the ledger. Measurements such as three-phase voltage and current, frequency, breaker status, protection scheme settings, network configuration settings (and other device configuration artifacts) and network traffic features (packet interarrival times) are compared every minute or other selected time windows. During phase 1 of the Grid Guard DLT project different DLT technologies were studies, and an assessment was performed on DLT technology vulnerabilities, uses, and key characteristics. DLT consensus protocols were studies (e.g., RAFT, named after Reliable, Replicated, Redundant, And Fault-Tolerant). Also, cryptography, public, private and permissioned or permissionless systems were assessed. Grid Guard implements a permissioned private DLT. Consensus algorithm selection and choice of DLT implementation depended heavily on the use-case. For this use-case, parameters were selected to measure performance and existing tools for assessment. Benchmarking was performed theoretically and practically. During phase 2 hashed transactions/blocks were inserted into the ledger every second. During phase 2 of the Grid Guard DLT project, a prototype framework was developed and demonstrated for attestation of critical substation devices and data using precision timing systems that use PTP and IRIG-B protocols) on a testbed of operational devices that emulated a distribution substation, control center, and power metering infrastructure using real Operational Technology (OT). The testbed includes OT devices such as protective relays, human machine interfaces (HMI), and power meters. To determine when to collect and compare system and network baselines, an initial examination of an anomaly detection capability to identify malicious manipulation of data streams was conducted. The resulting anomaly detection was demonstrated in a set of experiments and leveraged to trigger device artifact attestation checks. Attestation checks occur against device configuration baselines when compared with the immutable blockchain-stored baselines, which provided a cryptographically supported means by which to store baselines. The electrical substation-grid testbed was created to test the Grid Guard framework. The testbed emulates the operations of a portion of a power grid and SCADA systems as closely as possible. The testbed integrates real protocols, mainly IEC 61850 standard protocols, such as the Sampled Value (SV) and the GOOSE protocols. The testbed also supports DNP3 and other layer 2 and layer 3 protocols such as Telnet, SSH, SFTP/FTP and other proprietary protocols needed to connect to industrial control system equipment. The testbed emulates real power conditions using the OpalRT hardware-in-the-loop (HIL) device which can create fault situations that cannot be easily tested on real systems. The electrical substation-grid testbed was created using real measurement, communication, and protection devices that electrical utilities commonly use.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

Multiscale System Modeling of Single-Event-Induced Faults in Advanced Node Processors

Integration-technology feature shrink increases computing-system susceptibility to single-event effects (SEE). While modeling SEE faults will be critical, an integrated processor’s scope makes physically correct modeling computationally intractable. Without useful models, presilicon evaluation of fault-tolerance approaches becomes impossible. To incorporate accurate transistor-level effects at a system scope, we present a multiscale simulation framework. Charge collection at the 1) device level determines 2) circuit-level transient duration and state-upset likelihood. Circuit effects, in turn, impact 3) register-transfer-level architecture-state corruption visible at 4) the system level. Furthermore, the physically accurate effects of SEEs in large-scale systems, executed on a high-performance computing (HPC) simulator, could be used to drive cross-layer radiation hardening by design. We demonstrate the capabilities of this model with two case studies. First, we determine a D flip-flop’s sensitivity at the transistor level on 14-nm FinFet technology, validating the model against published cross sections. Second, we track and estimate faults in a microprocessor without interlocked pipelined stages (MIPS) processor for Adams 90% worst case environment in an isotropic space environment.

42 ENGINEERING↗
DOE OSTI2021

Open Source Fault-tolerant Grid Frequency Measurement for Solar Inverters

The Discrete Fourier transform (DFT) based measurement algorithms are one of the most common measurement algorithms for grid parameter estimation such as rms, phase angle, frequency. Over the past few years, many DFT based algorithms have been developed to enhance its measurement accuracy under steady-state and/or dynamic grid conditions. For example, an adaptive band-pass filter utilizing exponential modulation filter has been proposed to reduce measurement errors at the presence of large frequency deviations. Measurement accuracy of different algorithms including FIR filter, extended Kalman filtering (EKF), and enhanced DFT method have been compared in detail under different grid conditions. Two artificial signals that have 90-degree phase difference were constructed by the Clarke transformation to address the frequency spectrum leakage of DFT. A multi-module approach was developed to enhance both steady-state and dynamic measurement accuracies, in which each module was developed to eliminate some specific errors. Besides DFT-based measurement algorithms, some signal model-based algorithms have been developed to further improve the accuracy under dynamic conditions. However, a key drawback of the state-of-the-art algorithms is that they cannot perform measurements accurately during system transient faults. In the Blue Cut Fire event, there was a phase angle jump of about 26 degrees in the voltage waveform during the transient fault. The phase angle jump fault will cause waveform discontinuity, and these algorithms will fail to provide reliable measurements during this period because they typically assume the waveform to be measured is continuous, no matter what method (DFT, PLL, EKF, FIR, or Taylor WLS) is used for estimation. In fact, the measurement errors during the system transient faults like phase-jump is not required in the IEEE Standard. As a result, although a measurement instrument can pass the strict IEEE Standard, it could still be the source of the problem in the future if we have similar system transient faults, which could happen again. Therefore, developing the fault-tolerant measurement technology is the key to solve the problem.

14 SOLAR ENERGY↗
DOE OSTI2025

Fault Tolerant Decoding of QLDPC-GKP Codes with Circuit Level Soft Information

Concatenated bosonic-stabilizer codes have recently gained prominence as promising candidates for achieving low-overhead fault-tolerant quantum computing in the long term. In such systems, analog information obtained from the syndrome measurements of an inner bosonic code is used to inform decoding for an outer code layer consisting of a discrete-variable stabilizer code such as a surface code. The use of Quantum Low-Density Parity Check (QLDPC) codes as an outer code is of particular interest due to the significantly higher encoding rates offered by these code families, leading to a further reduction in overhead for large-scale quantum computing. Recent works have investigated the performance of QLDPC-GKP codes in detail, and the use of analog information from the inner code significantly boosts decoder performance. However, the noise models assumed in these works are typically limited to depolarizing or phenomenological noise. In this paper, we investigate the performance of QLDPC-GKP concatenated codes under circuit-level noise, based on a model introduced by Noh et al. in the context of the surface-GKP code. To demonstrate the performance boost from analog information, we investigate three scenarios: (a) decoding without soft information, (b) decoding with precomputed error probabilities but without real-time soft information, and (c) decoding with real-time soft information obtained from round-to-round decoding of the inner GKP code. Results show minimal improvement between (a) and (b), but a significant boost in (c), indicating that real-time soft information is critical for concatenated decoding under circuit-level noise. We also study the effect of measurement schedules with varying depths and show that using a schedule with minimum depth is essential for obtaining reliable soft information from the inner code.

Borah, Shantom K. [Arizona U. (main)]↗
DOE OSTI2024

ByzSec — A Multi-layered Byzantine Resilient Architecture for Bulk Power System Protective Relays

Reliability, selectivity, and sensitivity are the fundamental attributes of any protection system, acting as the main drivers in the selection of schemes, and equipment. In high-voltage systems, microprocessor-based relays represent the industry’s preferred solution, providing engineers with a vast array of benefits. However, they remain vulnerable to cybersecurity events that may compromise their functionality. To help mitigate against potential cybersecurity risks, this paper presents a fault-tolerant, Byzantine Resilient (BR) architecture that significantly increases the cybersecurity attributes of a protection system while minimizing the amount of performance impacts and integration overheads introduced. The solution relies on an array of independent relays that utilize robust consensus methods (based on Spire [1], [2]) to ensure correct system behavior is achieved even when a relay has been compromised. Furthermore, the solution has been complemented with a custom-built Situational Awareness engine that can be used to detect and identify potential threats. The implemented solution has been developed in consultation with three hardware vendors and has been tested to comply with the performance requirements of a 345kV differential protection scheme (87T). The results indicate that the proposed architecture is a comprehensive solution that: supports the strict correctness and performance requirements of the bulk power grid while providing a cost-effective alternative that offers a seamless, long-term solution.

byzantine security, Fault Tolerant Application Sof↗
DOE OSTI2026

Quantum-classical embedding via ghost Gutzwiller approximation for enhanced simulations of correlated electron systems

Simulating correlated materials on present-day quantum hardware remains challenging due to limited quantum resources. Quantum embedding methods offer a promising route by reducing computational complexity through the mapping of bulk systems onto effective impurity models, allowing more feasible simulations on pre- and early-fault-tolerant quantum devices. Here, this work develops a quantum-classical embedding framework based on the ghost Gutzwiller approximation to enable quantum-enhanced simulations of ground-state properties and spectral functions of correlated electron systems. Circuit complexity is analyzed using an adaptive variational quantum algorithm on a statevector simulator, applied to the infinite-dimensional Hubbard model with increasing ghost mode numbers from 3 to 5, resulting in circuit depths growing from 16 to 104. Noise effects are examined using a realistic error model, revealing significant impact on the spectral weight of the Hubbard bands. To mitigate these effects, the Iceberg quantum error detection code is employed, achieving up to 40% error reduction in simulations. Finally, the accuracy of the density matrix estimation and the derived spectral function is benchmarked on IBM and Quantinuum quantum hardware, featuring distinct qubit-connectivity and employing multiple levels of error mitigation techniques.

Chen, I-Chi [Ames Laboratory (AMES), Ames, IA (Uni↗
DOE OSTI2020

Algorithm-Based Fault Tolerance for Convolutional Neural Networks

Convolutional neural networks (CNNs) are becoming more and more important for solving challenging and critical problems in many fields. CNN inference applications have been deployed in safety-critical systems, which may suffer from soft errors caused by high-energy particles, high temperature, or abnormal voltage. Of critical importance is ensuring the stability of the CNN inference process against soft errors. Traditional fault tolerance methods are not suitable for CNN inference because error-correcting code is unable to protect computational components, instruction duplication techniques incur high overhead, and existing algorithm-based fault tolerance (ABFT) techniques cannot protect all convolution implementations. In this paper, we focus on how to protect the CNN inference process against soft errors as efficiently as possible, with the following three contributions. (1) We propose several systematic ABFT schemes based on checksum techniques and analyze their fault protection ability and runtime thoroughly. Unlike traditional ABFT based on matrix-matrix multiplication, our schemes support any convolution implementations. (2) We design a novel workflow integrating all the proposed schemes to obtain a high detection/correction ability with limited total runtime overhead. (3) We perform our evaluation using ImageNet with well-known CNN models including AlexNet, VGG-19, ResNet-18, and YOLOv2. Here, experimental results demonstrate that our implementation can handle soft errors with very limited runtime overhead (4%~8% in both error-free and error-injected situations).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Dependable classical-quantum computing systems engineering

Increasing evidence suggests quantum computing (QC) complements traditional High-Performance Computing (HPC) by leveraging its unique capabilities, leading to the emergence of a new, hybrid paradigm, QHPC. However, this integration introduces new challenges, with dependability–defined by reproducibility, resiliency, and security and privacy–emerging as a central concern for building trustworthy systems that provide an advantage to the users. This paper proposes a framework for dependable QHPC system design, organized around these three pillars. We identify integration challenges, anticipate roadblocks, and highlight productive synergies across QC, HPC, cloud platforms, and network security. Drawing from both classical computing principles and quantum-specific insights, we present a roadmap for co-design that supports robust hybrid architectures. Our approach offers concrete metrics for assessing dependability, provides design guidance for engineers working at the QC-HPC interface, and surfaces new engineering questions around complexity, scale, and fault tolerance. Ultimately, designing for dependability is key to realizing practical, scalable QHPC systems and accelerating the broader quantum ecosystem capable of translating quantum promises into actual application delivery.

HPC↗
DOE OSTI2021

Learning-based framework for sensor fault-tolerant building HVAC control with model-assisted learning

As people spend up to 87% of their time indoors, intelligent Heating, Ventilation, and Air Conditioning (HVAC) systems in buildings are essential for maintaining occupant comfort and reducing energy consumption. These HVAC systems in smart buildings rely 'on real-time sensor readings, which in practice often suffer from various faults and could also be vulnerable to malicious attacks. Such faulty sensor inputs may lead to the violation of indoor environment requirements (e.g., temperature, humidity, etc.) and the increase of energy consumption. While many model-based approaches have been proposed in the literature for building HVAC control, it is costly to develop accurate physical models for ensuring their performance and even more challenging to address the impact of sensor faults. In this work, we present a novel learning-based framework for sensor fault-tolerant HVAC control, which includes three deep learning based components for 1) generating temperature proposals with the consideration of possible sensor faults, 2) selecting one of the proposals based on the assessment of their accuracy, and 3) applying reinforcement learning with the selected temperature proposal. Moreover, to address the challenge of training data insufficiency in building-related tasks, we propose a model-assisted learning method leveraging an abstract model of building physical dynamics. Through extensive experiments, we demonstrate that the proposed fault-tolerant HVAC control framework can significantly reduce building temperature violations under a variety of sensor fault patterns while maintaining energy efficiency.

Xu, Shichao↗