Search NASA⌕ Search

SEARCH · Search NASA

Results for “Fault Injection”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Holistic Measurement Driven Resilience: Combining Operational Fault and Failure Measurements and Fault Injection for Quantifying Fault Detection, Propagation and Impact. Final report

For HPC systems to date, application resilience to faults and failures has been accomplished by the brute- force method of checkpoint/restart, which allows an application to make forward progress in the face of system and application faults, errors, and failures independent of root cause or end result. It has remained the primary resilience mechanism because we lack a way to identify faults and anticipate consequences early enough to take meaningful mitigating action. However, checkpoint/restart implementations put a tremendous burden on system resources and on the applications themselves and is becoming less feasible at scale. Because we have not yet operated at scales at which checkpoint/restart fails to provide forward progress, despite increasing costs, vendors have had little motivation to provide the instrumentation necessary for early identification of faults and failures. However, as we move from petascale to exascale, component mean time to failure (MTTF) will render the existing techniques ineffectual and/or too expensive. Furthermore, fault recovery mechanisms such as failover and/or error correction introduce performance inconsistency. Instrumentation allowing early indication of problems and tools to enable use of such information by systems, operating systems, and applications offer an alternative, more scalable and less costly solution. In the HMDR project, we built on our experience and expertise developed and accumulated over years of research on design, monitoring, measurement, and assessment of resilient computing systems. Analysis of field data on the current and past generations of extreme-scale systems revealed several challenges that, if not addressed in increasingly larger and more complex systems, may hinder the effectiveness of future exascale computing systems. Specifically, i) file systems and interconnects in current-generation large-scale systems already operate at the margins of resiliency, including consistent performance, and may not scale to larger deployments; ii) automated, software-based failover mechanisms are frequently inadequate and can introduce wider failures, such that failures during recovery may lead to system/application failures, including system-wide outages; and iii) silent data corruption represents a critical fault mode and will require efficient detection mechanisms if next-generation applications are to take full advantage of exascale hardware. To address the above challenges, we assembled a team of world-renowned experts in resilient extreme- scale computing from the University of Illinois (Electrical and Computer Engineering, Computer Science, and NCSA), SNL, LANL, NERSC, and Cray. Our team includes representatives from centers that house many of the largest HPC resources in the world, both today and over the coming years. The team has a unique track record of research in i) system and application failure characterization based on the analysis of field data, ii) data-driven design of fault/error detection mechanisms, and iii) experimental characterization of system/application resiliency. The team includes system owners/operators who provide continuous data collection and access and ensure installation of appropriate analysis tools.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

A Cooperative Approach to Virtual Machine Based Fault Injection

Resilience investigations often employ fault injection (FI) tools to study the effects of simulated errors on a target system. It is important to keep the target system under test (SUT) isolated from the controlling environment in order to maintain control of the experiement. Virtual machines (VMs) have been used to aid these investigations due to the strong isolation properties of system-level virtualization. A key challenge in fault injection tools is to gain proper insight and context about the SUT. In VM-based FI tools, this challenge of target con- text is increased due to the separation between host and guest (VM). We discuss an approach to VM-based FI that leverages virtual machine introspection (VMI) methods to gain insight into the target s context running within the VM. The key to this environment is the ability to provide basic information to the FI system that can be used to create a map of the target environment. We describe a proof- of-concept implementation and a demonstration of its use to introduce simulated soft errors into an iterative solver benchmark running in user-space of a guest VM.

Naughton, III, Thomas J↗
DOE OSTI2019

SAFIRE : Scalable and Accurate Fault Injection for Parallel Multi-threaded Applications

The software SAFIRE implements a fault injection tool to evaluate the impact of soft errors on parallel, multi-threaded applications. It implements instruction-level fault models that emulate soft errors by flipping bits in the register operands of executed machine instructions. SAFIRE is based on the LLVM compiler and utilizes novel techniques to optimize instruction instrumentation and fault injection. By detailed measurements, SAFIRE is multiple times faster than other state-of-the-art fault injection techniques, while having the same accuracy.

Georgakoudis, Giorgis↗
DOE OSTI2022

Fault Injection for TensorFlow Applications

As machine learning (ML) has seen increasing adoption in safety-critical domains (e.g., autonomous vehicles), the reliability of ML systems has also grown in importance. While prior studies have proposed techniques to enable efficient error-resilience (e.g., selective instruction duplication), a fundamental requirement for realizing these techniques is a detailed understanding of the application’s resilience. In this work, we present TensorFI 1 and TensorFI 2, high-level fault injection (FI) frameworks for TensorFlow-based applications. TensorFI 1 and 2 are able to inject both hardware and software faults in any general TensorFlow 1 and 2 program respectively. Both are configurable FI tools that are flexible, easy to use, and portable. They can be integrated into existing TensorFlow programs to assess their resilience for different fault types (e.g., bit-flips in particular operations or layers). We use the TensorFI 1 and TensorFI 2 to evaluate the resilience of 12 and 10 ML programs written in TensorFlow, including DNNs used in the autonomous vehicle domain. The results give us insights into why some of the models are more resilient. We also measure the performance overheads of the two injectors, and present 4 case studies, two for each tool, to demonstrate their utility.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

CO2 Push-Pull Single Fault Injection Simulations

ASCII text files containing grid-block name, X-Y-Z location, and multiple parameters from TOUGH2 simulation output of CO2 injection into an idealized single fault representing a dipping normal fault at the Desert Peak geothermal field (readable by GMS). The fault is composed of a damage zone, a fault gouge and a slip plane. The runs are described in detail in the following: Borgia A., Oldenburg C.M., Zhang R., Jung Y., Lee K.J., Doughty C., Daley T.M., Chugunov N., Altundas B, Ramakrishnan T.S., 2017. Carbon Dioxide Injection for Enhanced Characterization of Faults and Fractures in Geothermal Systems. Proceedings of the 42st Workshop on Geothermal Reservoir Engineering, Stanford University, Stanford, California, February 13-17.

15 GEOTHERMAL ENERGY↗
DOE OSTI2021

Information Leakage Analysis using a Co-design-Based Fault Injection Technique on a RISC-V Microprocessor

The RISC-V instruction set architecture open licensing policy has spawned a hive of development activity, making a range of implementations publicly available. The environments in which RISC-V operates have expanded correspondingly, driving the need for a generalized approach to evaluating the reliability of RISC-V implementations under adverse operating conditions or after normal wear-out periods. Fault injection (FI) refers to the process of changing the state of registers or wires, either permanently or momentarily, and then observing execution behavior. The analysis provides insight into the development of countermeasures that protect against the leakage or corruption of sensitive information which might occur because of unexpected execution behavior. In this paper, we develop a hardware-software co-design architecture that enables fast, configurable fault emulation and utilize it for information leakage and data corruption analysis. Modern System-on-chip FPGAs enable building an evaluation platform where control elements run on a processor(s) (PS) simultaneously with the target design running in the programmable logic (PL). Software components of the FI system introduce faults and report execution behavior. A pair of RISC-V FI-instrumented implementations are created and configured to execute the Advanced Encryption Standard and Twister algorithms. Key and plaintext information leakage and degraded pseudo-random sequences are both observed in the output for a subset of the emulated faults.

42 ENGINEERING↗
DOE OSTI2018

Characterizing the Impact of Soft Errors Affecting Floating-point ALUs using RTL-level Fault Injection

Strategies to detect, correct, or mitigate the impact of soft errors rely on errors-injection experiments. For efficient evaluation, such experiments typically inject errors in software by sampling errors from a candidate distribution. Most often, these strategies randomly select and flip one bit in the output of an instruction. While single-bit flips might constitute a meaningful model for errors affecting hardware, the appropriateness of this model for software-based errors has not been studied. In this paper, we study the manifestation of errors in the output registers due to errors affecting candidate instructions executed by floating point ALUs. We inject single-bit flips into the RTL descriptions of floating point ALUs and analyze the differences between anticipated and observed outputs when executing floating-point addition, subtraction, multiplication, and division. We choose the operands for these instructions randomly and from operands observed in five benchmarks. We observe a rich distribution of errors in the output and analyze their implications for software-based fault-injection campaigns.

Subasi, Omer↗
DOE OSTI2017

CO2 Push-Pull Dual (Conjugate) Faults Injection Simulations

This submission contains datasets and a final manuscript associated with a project simulating carbon dioxide push-pull into a conjugate fault system modeled after Dixie Valley- sensitivity analysis of significant parameters and uncertainty prediction by data-worth analysis. Datasets include: (1) Forward simulation runs of standard cases (push & pull phases), (2) Local sensitivity analyses (push & pull phases), and (3) Data-worth analysis (push & pull phases).

15 GEOTHERMAL ENERGY↗
DOE OSTI2019

Mathematical Modelling of Fault Reactivation Induced by Water Injection

Faults in the host rock that might exist in the vicinity of deep geological repositories for radioactive waste, constitute potential enhanced pathways for radionuclide migration. Several processes might trigger pore pressure increases in the faults leading to fault failure and induced seismicity, and increase the faults’ permeability. In this research, we developed a mathematical model to simulate fault activation during an experiment of controlled water injection in a fault at the Mont-Terri Underground Research Laboratory in Switzerland. The effects of in-situ stress, fault shear strength parameters and heterogeneity are assessed. It was shown that the above factors are critical and need to be adequately characterized in order to predict the faults’ hydro-mechanical behaviour.

58 GEOSCIENCES↗
DOE OSTI2024

Understanding Mixed Precision GEMM with MPGemmFI: Insights into Fault Resilience

Emerging deep learning workloads urgently need fast general matrix multiplication (GEMM). Thus, one of the critical features of machine-learning-specific accelerators such as NVIDIA Tensor Cores, AMD Matrix Cores, and Google TPUs is the support of mixed-precision enabled GEMM. For DNN models, lower-precision FP data formats and computation offer acceptable correctness but significant performance, area, and memory footprint improvement. While promising, the mixed-precision computation on error resilience remains unexplored. To this end, we develop a fault injection framework that systematically injects fault into the mixed-precision computation results. We investigate how the faults affect the accuracy of machine learning applications. Based on the characteristics of error resilience, we offer lightweight error detection and correction solutions that significantly improve the overall model accuracy by 75% if the models experience hardware faults. The solutions can be efficiently integrated into the accelerator's pipelines.

Fang, Bo↗
DOE OSTI2017

Recent results from a testbed for the reliability of the digital instrumentation and control systems and cybersecurity in nuclear power plants - 392

Transition from analog to digital instrumentation and control systems requires a systematic study of the modifications, and assessment of their reliability. Such studies are often carried out on a test bed. The test bed developed earlier includes a commercially available triple modular redundant (TMR) programmable logic controller (PLC) for testing not only the reliability of the digital I and C but also the cyber-security of the system during plant operation. This paper reports results of our preliminary analyses carried out using the test bed following the injection of faults mimicking faulty sensors. The fault injection tool generates faulty signals in single or multiple sensors located in the primary loop. These scenarios studied under steady-state operation conditions simulate a bad sensor, a cut wire, noise, etc. The faults injected can lead to information being displayed in the main control room (MCR) to not match the conditions in the plant. In addition to a fault injection module, a result analyzer module is also a part of this testbed. Operator actions can be based on information displayed in the MCR, or they can be informed by the result analyzer. Potential operator actions following these single- or multiple faulty signals are assessed. It is concluded that the result analyzer in some cases can be helpful in guiding operators to make the right decisions. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2017

Characterizing the Potential for Injection-Induced Fault Reactivation Through Subsurface Structural Mapping and Stress Field Analysis, Wellington Field, Sumner County, Kansas: Injection-Induced Fault Slip, Kansas

Kansas, like other parts of the central U.S., has experienced a recent increase in seismicity. Correlation of these events with brine disposal operations suggests pore fluid pressure increases are reactivating preexisting faults, but rigorous evaluation at injection sites is lacking. Here we determine the suitability of CO 2 injection into the Cambrian-Ordovician Arbuckle Group for long-term storage and into a Mississippian reservoir for enhanced oil recovery in Wellington Field, Sumner County, Kansas. To determine the potential for injection-induced earthquakes, we map subsurface faults and estimate in situ stresses, perform slip and dilation tendency analyses to identify well-oriented faults relative to the estimated stress field, and determine the pressure changes required to induce slip at reservoir and basement depths. Three-dimensional seismic reflection data reveal 12 near-vertical faults, mostly striking NNE, consistent with nodal planes from moment tensor solutions from recent earthquakes in the region. Most of the faults cut both reservoirs and several clearly penetrate the Precambrian basement. Drilling-induced fractures (N = 40) identified from image logs and inversion of earthquake moment tensor solutions (N = 65) indicate that the maximum horizontal stress is approximately EW. Slip tendency analysis indicates that faults striking <020° are stable under current reservoir conditions, whereas faults striking 020°–049° may be prone to reactivation with increasing pore fluid pressure. Although the proposed injection volume (40,000 t) is unlikely to reactive faults at reservoir depths, high-rate injection operations could reach pressures beyond the critical threshold for slip within the basement, as demonstrated by the large number of injection-induced earthquakes west of the study area.

54 ENVIRONMENTAL SCIENCES↗