Engineering topics

Iyer, Ravishankar K.

Publications and source records attributed to Iyer, Ravishankar K..

At least 19 records

NASA NTRS1997

Fault Injection Techniques and Tools

Dependability evaluation involves the study of failures and errors. The destructive nature of a crash and long error latency make it difficult to identify the causes of failures in the operational environment. It is particularly hard to recreate a failure scenario for a large, complex system. To identify and understand potential failures, we use an experiment-based approach for studying the dependability of a system. Such an approach is applied not only during the conception and design phases, but also during the prototype and operational phases. To take an experiment-based approach, we must first understand a system's architecture, structure, and behavior. Specifically, we need to know its tolerance for faults and failures, including its built-in detection and recovery mechanisms, and we need specific instruments and tools to inject faults, create failures or errors, and monitor their effects.

Hsueh, Mei-Chen↗

NASA NTRS1995

Measuring fault tolerance with the FTAPE fault injection tool

This paper describes FTAPE (Fault Tolerance And Performance Evaluator), a tool that can be used to compare fault-tolerant computers. The major parts of the tool include a system-wide fault-injector, a workload generator, and a workload activity measurement tool. The workload creates high stress conditions on the machine. Using stress-based injection, the fault injector is able to utilize knowledge of the workload activity to ensure a high level of fault propagation. The errors/fault ratio, performance degradation, and number of system crashes are presented as measures of fault tolerance.

Tsai, Timothy K.↗

NASA NTRS1995

FTAPE: A fault injection tool to measure fault tolerance

The paper introduces FTAPE (Fault Tolerance And Performance Evaluator), a tool that can be used to compare fault-tolerant computers. The tool combines system-wide fault injection with a controllable workload. A workload generator is used to create high stress conditions for the machine. Faults are injected based on this workload activity in order to ensure a high level of fault propagation. The errors/fault ratio and performance degradation are presented as measures of fault tolerance.

Tsai, Timothy K.↗

NASA NTRS1995

Software dependability in the Tandem GUARDIAN system

Based on extensive field failure data for Tandem's GUARDIAN operating system this paper discusses evaluation of the dependability of operational software. Software faults considered are major defects that result in processor failures and invoke backup processes to take over. The paper categorizes the underlying causes of software failures and evaluates the effectiveness of the process pair technique in tolerating software faults. A model to describe the impact of software faults on the reliability of an overall system is proposed. The model is used to evaluate the significance of key factors that determine software dependability and to identify areas for improvement. An analysis of the data shows that about 77% of processor failures that are initially considered due to software are confirmed as software problems. The analysis shows that the use of process pairs to provide checkpointing and restart (originally intended for tolerating hardware faults) allows the system to tolerate about 75% of reported software faults that result in processor failures. The loose coupling between processors, which results in the backup execution (the processor state and the sequence of events) being different from the original execution, is a major reason for the measured software fault tolerance. Over two-thirds (72%) of measured software failures are recurrences of previously reported faults. Modeling, based on the data, shows that, in addition to reducing the number of software faults, software dependability can be enhanced by reducing the recurrence rate.

Lee, Inhwan↗

NASA NTRS1994

Software fault tolerance in computer operating systems

This chapter provides data and analysis of the dependability and fault tolerance for three operating systems: the Tandem/GUARDIAN fault-tolerant system, the VAX/VMS distributed system, and the IBM/MVS system. Based on measurements from these systems, basic software error characteristics are investigated. Fault tolerance in operating systems resulting from the use of process pairs and recovery routines is evaluated. Two levels of models are developed to analyze error and recovery processes inside an operating system and interactions among multiple instances of an operating system running in a distributed environment. The measurements show that the use of process pairs in Tandem systems, which was originally intended for tolerating hardware faults, allows the system to tolerate about 70% of defects in system software that result in processor failures. The loose coupling between processors which results in the backup execution (the processor state and the sequence of events occurring) being different from the original execution is a major reason for the measured software fault tolerance. The IBM/MVS system fault tolerance almost doubles when recovery routines are provided, in comparison to the case in which no recovery routines are available. However, even when recovery routines are provided, there is almost a 50% chance of system failure when critical system jobs are involved.

Iyer, Ravishankar K.↗

NASA NTRS1994

FTAPE: A fault injection tool to measure fault tolerance

Tsai, Timothy K.↗

NASA NTRS1993

Prediction-based dynamic load-sharing heuristics

The authors present dynamic load-sharing heuristics that use predicted resource requirements of processes to manage workloads in a distributed system. A previously developed statistical pattern-recognition method is employed for resource prediction. While nonprediction-based heuristics depend on a rapidly changing system status, the new heuristics depend on slowly changing program resource usage patterns. Furthermore, prediction-based heuristics can be more effective since they use future requirements rather than just the current system state. Four prediction-based heuristics, two centralized and two distributed, are presented. Using trace driven simulations, they are compared against random scheduling and two effective nonprediction based heuristics. Results show that the prediction-based centralized heuristics achieve up to 30 percent better response times than the nonprediction centralized heuristic, and that the prediction-based distributed heuristics achieve up to 50 percent improvements relative to their nonprediction counterpart.

Goswami, Kumar K.↗

NASA NTRS1993

Computing in space: Issues and answers

Future plans for space exploration call for scientific instruments whose data is increasingly voluminous, far exceeding constraints set by telemetry rates, and therefore, an increased role for spaceborne computing. Three issues that greatly influence the realization of spaceborne computing systems are addressed. The first involves the identification of those research problems which are unique to space computing. Resources must be expended on these problems if efforts are to add value to, and have impact on, the technology of computing in space. The second addresses the transfer of new technology from the research laboratory to use in space. What can be done to expedite this process and shrink the technology gap that separates Earth and space based systems must be assessed. The third issue involves the user's point of view of what is required of new space computing systems. That is, what are the driving forces that most influence customers' decisions to use or not use new technology? Issues of design environment for spaceborne computers, tools for modeling and evaluation of flight systems, and validation of space flight hardware and software are also discussed.

Davidson, John M.↗

NASA NTRS1993

Modeling the data management system of Space Station Freedom with DEPEND

Some of the features and capabilities of the DEPEND simulation-based modeling tool are described. A study of a 1553B local bus subsystem of the Space Station Freedom Data Management System (SSF DMS) is used to illustrate some types of system behavior that can be important to reliability and performance evaluations of this type of spacecraft. A DEPEND model of the subsystem is used to illustrate how these types of system behavior can be modeled, and shows what kinds of engineering and design questions can be answered through the use of these modeling techniques. DEPEND's process-based simulation environment is shown to provide a flexible method for modeling complex interactions between hardware and software elements of a fault-tolerant computing system.

Olson, Daniel P.↗

NASA NTRS1992

FOCUS - An experimental environment for fault sensitivity analysis

FOCUS, a simulation environment for conducting fault-sensitivity analysis of chip-level designs, is described. The environment can be used to evaluate alternative design tactics at an early design stage. A range of user specified faults is automatically injected at runtime, and their propagation to the chip I/O pins is measured through the gate and higher levels. A number of techniques for fault-sensitivity analysis are proposed and implemented in the FOCUS environment. These include transient impact assessment on latch, pin and functional errors, external pin error distribution due to in-chip transients, charge-level sensitivity analysis, and error propagation models to depict the dynamic behavior of latch errors. A case study of the impact of transient faults on a microprocessor-based jet-engine controller is used to identify the critical fault propagation paths, the module most sensitive to fault propagation, and the module with the highest potential for causing external errors.

Choi, Gwan S.↗

NASA NTRS1992

DEPEND: A simulation-based environment for system level dependability analysis

The design and evaluation of highly reliable computer systems is a complex issue. Designers mostly develop such systems based on prior knowledge and experience and occasionally from analytical evaluations of simplified designs. A simulation-based environment called DEPEND which is especially geared for the design and evaluation of fault-tolerant architectures is presented. DEPEND is unique in that it exploits the properties of object-oriented programming to provide a flexible framework with which a user can rapidly model and evaluate various fault-tolerant systems. The key features of the DEPEND environment are described, and its capabilities are illustrated with a detailed analysis of a real design. In particular, DEPEND is used to simulate the Unix based Tandem Integrity fault-tolerance and evaluate how well it handles near-coincident errors caused by correlated and latent faults. Issues such as memory scrubbing, re-integration policies, and workload dependent repair times which affect how the system handles near-coincident errors are also evaluated. Issues such as the method used by DEPEND to simulate error latency and the time acceleration technique that provides enormous simulation speed up are also discussed. Unlike any other simulation-based dependability studies, the use of these approaches and the accuracy of the simulation model are validated by comparing the results of the simulations, with measurements obtained from fault injection experiments conducted on a production Tandem Integrity machine.

Goswami, Kumar↗

NASA NTRS1990

Simulated fault injection - A methodology to evaluate fault tolerant microprocessor architectures

A simulation-based fault-injection method for validating fault-tolerant microprocessor architectures is described. The approach uses mixed-mode simulation (electrical/logic analysis), and injects transient errors in run-time to assess the resulting fault impact. As an example, a fault-tolerant architecture which models the digital aspects of a dual-channel real-time jet-engine controller is used. The level of effectiveness of the dual configuration with respect to single and multiple transients is measured. The results indicate 100 percent coverage of single transients. Approximately 12 percent of the multiple transients affect both channels; none result in controller failure since two additional levels of redundancy exist.

Choi, Gwan S.↗

NASA NTRS1990

Dynamic load-sharing using predicted process resource requirements

Heuristics which use predicted process resource requirements to make scheduling decisions are proposed. Four heuristics are presented. The first two, MINQ and SMPL, employ centralized scheduling and the remaining two, DMINQ and FDMINQ, use distributed scheduling. These heuristics are first compared against random scheduling and then against two conventional heuristics, CENTEX and DISTED, which schedule processes solely based on system state information. Results based on trace-driven simulations show that the proposed centralized heuristics offer significantly improved mean response time and they require fewer status update messages. In experiments using the same status update rates, SMPL response times were, on the average, 22 percent lower than those for CENTEX; MINQ response times were, on the average, 18 percent lower. The simulations also showed that MINQ and SMPL can perform as well as, or better than, CENTEX while using up to 70 percent fewer status update messages. The use of fewer status update messages imposes less overhead on the system. The use of prediction for distributed scheduling produced similar results. When prediction was used to filter small processes and execute them locally a 50 percent improvement in response times was obtained.

Goswami, Kumar K.↗

NASA NTRS1990

Failure analysis and modeling of a VAXcluster system

This paper discusses the results of a measurement-based analysis of real error data collected from a DEC VAXcluster multicomputer system. In addition to evaluating basic system dependability characteristics such as error and failure distributions and hazard rates for both individual machines and for the VAXcluster, reward models were developed to analyze the impact of failures on the system as a whole. The results show that more than 46 percent of all failures were due to errors in shared resources. This is despite the fact that these errors have a recovery probability greater than 0.99. The hazard rate calculations show that not only errors, but also failures occur in bursts. Approximately 40 percent of all failures occur in bursts and involved multiple machines. This result indicates that correlated failures are significant. Analysis of rewards shows that software errors have the lowest reward (0.05 vs 0.74 for disk errors). The expected reward rate (reliability measure) of the VAXcluster drops to 0.5 in 18 hours for the 7-out-of-7 model and in 80 days for the 3-out-of-7 model.

Tang, Dong↗

NASA NTRS1990

DSIM: A distributed simulator

Discrete event-driven simulation makes it possible to model a computer system in detail. However, such simulation models can require a significant time to execute. This is especially true when modeling large parallel or distributed systems containing many processors and a complex communication network. One solution is to distribute the simulation over several processors. If enough parallelism is achieved, large simulation models can be efficiently executed. This study proposes a distributed simulator called DSIM which can run on various architectures. A simulated test environment is used to verify and characterize the performance of DSIM. The results of the experiments indicate that speedup is application-dependent and, in DSIM's case, is also dependent on how the simulation model is distributed among the processors. Furthermore, the experiments reveal that the communication overhead of ethernet-based distributed systems makes it difficult to achieve reasonable speedup unless the simulation model is computation bound.

Goswami, Kumar K.↗

NASA NTRS1990

Automatic recognition of intermittent failures - An experimental study of field data

A methodology is proposed for recognizing the symptoms of persistent problems in large systems. The system error rate is used to identify the error states among which relationships may exist. Statistical techniques are used to validate and quantify the strength of the relationship among these error states. As input, the approach takes the raw error logs containing a single entry for each error that is detected as an isolated event. As output, it produces a list of symptoms that characterize persistent errors. Thus, given a failure, it is determined whether the failure is an intermittent manifestation of a common fault or whether it is an isolated (transient) incident. The technique is shown to work on two CYBER systems and an IBM 3081 multiprocessor system. Comparisons to real failure/repair information obtained from field engineers show that, in about 85 percent of the cases, the error symptoms recognized by this approach correspond to real problems. The remaining 15 percent of the cases, although not directly supported by field data, are confirmed as being valid problems.

Iyer, Ravishankar K.↗

NASA NTRS1990

Analysis of field data on computer failures

This paper emphasizes the importance of making field measurements for effective and realistic dependability evaluations. Two examples are given, both based on real data from IBM mainframes. The first evaluates the impact of the operating environment on system failure characteristics and the second shows how an accurate model depicting this interaction can be extracted from real data.

Iyer, Ravishankar K.↗

NASA NTRS1990

DEPEND - A design environment for prediction and evaluation of system dependability

The development of DEPEND, an integrated simulation environment for the design and dependability analysis of fault-tolerant systems, is described. DEPEND models both hardware and software components at a functional level, and allows automatic failure injection to assess system performance and reliability. It relieves the user of the work needed to inject failures, maintain statistics, and output reports. The automatic failure injection scheme is geared toward evaluating a system under high stress (workload) conditions. The failures that are injected can affect both hardware and software components. To illustrate the capability of the simulator, a distributed system which employs a prediction-based, dynamic load-balancing heuristic is evaluated. Experiments were conducted to determine the impact of failures on system performance and to identify the failures to which the system is especially susceptible.

Goswami, Kumar K.↗