SEARCH · Search NASA

Results for “root cause diagnosis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

All sources NASA NTRS DOE OSTI

DOE OSTI2012

3-Dimensional Root Cause Diagnosis via Co-analysis

With the growth of system size and complexity, reliability has become a major concern for large-scale systems. Upon the occurrence of failure, system administrators typically trace the events in Reliability, Availability, and Serviceability (RAS) logs for root cause diagnosis. However, RAS log only contains limited diagnosis information. Moreover, the manual processing is time-consuming, error-prone, and not scalable. To address the problem, in this paper we present an automated root cause diagnosis mechanism for large-scale HPC systems. Our mechanism examines multiple logs to provide a 3-D fine-grained root cause analysis. Here, 3-D means that our analysis will pinpoint the failure layer, the time, and the location of the event that causes the problem. We evaluate our mechanism by means of real logs collected from a production IBM Blue Gene/P system at Oak Ridge National Laboratory. It successfully identifies failure layer information for 219 failures during 23-month period. Furthermore, it effectively identifies the triggering events with time and location information, even when the triggering events occur hundreds of hours before the resulting failures.

Zheng, Ziming↗

DOE OSTI2012

3-Dimensional root cause diagnosis via co-analysis. In: ICAC '12 Proceedings of the 9th international conference on Autonomic computing

Zheng, Ziming↗

NASA NTRS2012

Model-Based Fault Diagnosis: Performing Root Cause and Impact Analyses in Real Time

Generic, object-oriented fault models, built according to causal-directed graph theory, have been integrated into an overall software architecture dedicated to monitoring and predicting the health of mission- critical systems. Processing over the generic fault models is triggered by event detection logic that is defined according to the specific functional requirements of the system and its components. Once triggered, the fault models provide an automated way for performing both upstream root cause analysis (RCA), and for predicting downstream effects or impact analysis. The methodology has been applied to integrated system health management (ISHM) implementations at NASA SSC's Rocket Engine Test Stands (RETS).

Figueroa, Jorge F.↗

NASA NTRS1992

Intelligent alarming

This talk discusses the importance of providing a process operator with concise information about a process fault including a root cause diagnosis of the problem, a suggested best action for correcting the fault, and prioritization of the problem set. A decision tree approach is used to illustrate one type of approach for determining the root cause of a problem. Fault detection in several different types of scenarios is addressed, including pump malfunctions and pipeline leaks. The talk stresses the need for a good data rectification strategy and good process models along with a method for presenting the findings to the process operator in a focused and understandable way. A real time expert system is discussed as an effective tool to help provide operators with this type of information. The use of expert systems in the analysis of actual versus predicted results from neural networks and other types of process models is discussed.

Braden, W. B.↗

DOE OSTI2012

Refurbishment and Testing of the 1970's Era LASS Solenoid Coils for JLab's Hall D

JLab refurbished the LASS1, 1.85 m bore Solenoid, consisting of four superconducting coils to act as the principal analysis magnet for nuclear physics in the newly constructed, Hall D at Jefferson Lab. The coils, built in 1971 at Stanford Linier Accelerator Center and used a second time at the MEGA Experiment at Los Alamos, had electrical shorts and leaks to the insulating vacuum along with deteriorated superinsulation & instrumentation. Root cause diagnosis of the problems and the repair methods are described along with the measures used to qualify the vessels and piping within the Laboratory's Pressure Safety Program (mandated by 10CFR851). The extraordinary refrigerator operational methods used to utilize the obsolete cryogenic apparatus gathered for the off-line, single coil tests are described.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2021

Hardware-in-the-Loop Testbed for Cyber-Physical Security of Photovoltaic Farms

In the last decades, modem grids with distributed energy resources, such as photovoltaic (PV) farms, are increasingly vulnerable to cyber-attacks that seriously affect the stability and performance of the power system. While cyber-physical security of smart grids is extensively studied, most of the existing work focuses on the grid level and neglects the modeling and features of device-level power electronics converters (PECs). Furthermore, establishing a high-fidelity simulation testbed that can simulate harmonic frequencies of the PV farm is in urgent need. In this paper, a high-fidelity and real-time hardware-in- the-loop testbed is built to simulate the harmonics of power electronics converters for cyber-physical security of PEC-enabled PV farms. Based on this testbed, the impact of typical cyber-attacks and physical faults on the PV converter can be analyzed, thus providing a foundation for cyber-attack detection, root cause diagnosis, and resilient control to mitigate the adverse effects of cyber-attacks.

14 SOLAR ENERGY↗

NASA NTRS2008

Operator Performance Evaluation of Fault Management Interfaces for Next-Generation Spacecraft

In the cockpit of the NASA's next generation of spacecraft, most of vehicle commanding will be carried out via electronic interfaces instead of hard cockpit switches. Checklists will be also displayed and completed on electronic procedure viewers rather than from paper. Transitioning to electronic cockpit interfaces opens up opportunities for more automated assistance, including automated root-cause diagnosis capability. The paper reports an empirical study evaluating two potential concepts for fault management interfaces incorporating two different levels of automation. The operator performance benefits produced by automation were assessed. Also, some design recommendations for spacecraft fault management interfaces are discussed.

Hayashi, Miwa↗

DOE OSTI2012

Filtering log data: finding the needles in the haystack

Log data is an incredible asset for troubleshooting in large-scale systems. Nevertheless, due to the ever-growing system scale, the volume of such data becomes overwhelming, bringing enormous burdens on both data storage and data analysis. To address this problem, we present a 2-dimensional online filtering mechanism to remove redundant and noisy data via feature selection and instance selection. The objective of this work is two-fold: (i) to significantly reduce data volume without losing important information, and (ii) to effectively promote data analysis. We evaluate this new filtering mechanism by means of real environmental data from the production supercomputers at Oak Ridge National Laboratory and Sandia National Laboratory. Our preliminary results demonstrate that our method can reduce more than 85% disk space, thereby significantly reducing analysis time. Moreover, it also facilitates better failure prediction and diagnosis by more than 20%, as compared to the conventional predictive approach relying on RAS (Reliability, Availability, and Serviceability) events alone.

failure prediction↗

NASA NTRS2012

NASA Tech Briefs, February 2012

This issue contains the following briefs: (1) Optical Comb from a Whispering Gallery Mode Resonator for Spectroscopy and Astronomy Instruments Calibration (2) Real-Time Flight Envelope Monitoring System (3) Nemesis Autonomous Test System (4) Mirror Metrology Using Nano-Probe Supports (5) Automated Lab-on-a-Chip Electrophoresis System (6) Techniques for Down-Sampling a Measured Surface Height Map for Model Validation (7) Multi-Component, Multi-Point Interferometric Rayleigh/Mie Doppler Velocimeter (8) Frequency to Voltage Converter Analog Front-End Prototype (9) Dust-Tolerant Intelligent Electrical Connection System (10) Gigabit Ethernet Asynchronous Clock Compensation FIFO (11) High-Speed, Multi-Channel Serial ADC LVDS Interface for Xilinx Virtex-5 FPGA (12) Glovebox for GeoLab Subsystem in HDU1-PEM (13) Modified Process Reduces Porosity when Soldering in Reduced Gravity Environments (14) Use of Functionalized Carbon Nanotubes for Covalent Attachment of Nanotubes to Silicon (15) Flexible Plug Repair for Shuttle Wing Leading Edge (16) Three Dimensionally Interlinked, Dense, Solid Form of Single-Walled CNT Ropes (17) Axel Robotic Platform for Crater and Extreme Terrain Exploration (18) Site Tamper and Material Plow Tool - STAMP (19) Magnetic Interface for Segmented Mirror Assembly (20) Transpiration-Cooled Spacecraft-Insulation-Repair Fasteners (21) Fluorescence-Based Sensor for Monitoring Activation of Lunar Dust (22) Aperture Ion Source (23) Virtual Ultrasound Guidance for Inexperienced Operators (24) Model-Based Fault Diagnosis: Performing Root Cause and Impact Analyses in Real Time (25) Interactive Schematic Integration Within the Propellant System Modeling Environment (26) Magnetic and Electric Field Polarizations of Oblique Magnetospheric Chorus Waves (27) Variable Sampling Mapping.

Source record↗

NASA NTRS2006

ISHM Decision Analysis Tool: Operations Concept

The state-of-the-practice Shuttle caution and warning system warns the crew of conditions that may create a hazard to orbiter operations and/or crew. Depending on the severity of the alarm, the crew is alerted with a combination of sirens, tones, annunciator lights, or fault messages. The combination of anomalies (and hence alarms) indicates the problem. Even with much training, determining what problem a particular combination represents is not trivial. In many situations, an automated diagnosis system can help the crew more easily determine an underlying root cause. Due to limitations of diagnosis systems,however, it is not always possible to explain a set of alarms with a single root cause. Rather, the system generates a set of hypotheses that the crew can select from. The ISHM Decision Analysis Tool (IDAT) assists with this task. It presents the crew relevant information that could help them resolve the ambiguity of multiple root causes and determine a method for mitigating the problem. IDAT follows graphical user interface design guidelines and incorporates a decision analysis system. I describe both of these aspects.

Source record↗

DOE OSTI2022

Battery aging mode identification across NMC compositions and designs using machine learning

A comprehensive understanding of lithium-ion battery (LiB) lifespan is the key to designing durable batteries and optimizing use protocols. Although battery lifetime prediction methods are flourishing, diagnosis of the root causes of aging and degradation have not yet been well developed nor studied for a broad mixture of designs and use cases. Here, we create a machine-learning (ML)-based framework that distinguishes aging modes using multiple electrochemical signatures recorded cycle-by-cycle. The predominant aging behaviors include a combination of loss of active materials in cathode (LAMPE) and a loss of Li inventory (LLI) in Li plating or solid electrolyte interphase (SEI) formation, manifested from 44 batteries representing two cathode chemistries, two electrode loadings, and five charging rates. Here, the aging mode classification accuracy is 86% using features within the first 50 cycles and increases to 88% beyond 225 cycles. The same features can quantify the percentage of end-of-life LAMPE with only 4.3% of error.

25 ENERGY STORAGE↗

NASA NTRS2011

Integrated Software Health Management for Aircraft GN and C

Modern aircraft rely heavily on dependable operation of many safety-critical software components. Despite careful design, verification and validation (V&V), on-board software can fail with disastrous consequences if it encounters problematic software/hardware interaction or must operate in an unexpected environment. We are using a Bayesian approach to monitor the software and its behavior during operation and provide up-to-date information about the health of the software and its components. The powerful reasoning mechanism provided by our model-based Bayesian approach makes reliable diagnosis of the root causes possible and minimizes the number of false alarms. Compilation of the Bayesian model into compact arithmetic circuits makes SWHM feasible even on platforms with limited CPU power. We show initial results of SWHM on a small simulator of an embedded aircraft software system, where software and sensor faults can be injected.

Schumann, Johann↗

DOE OSTI2015

Monitoring Extreme-scale Lustre Toolkit

We discuss the design and ongoing development of the Monitoring Extreme-scale Lustre Toolkit (MELT), a unified Lustre performance monitoring and analysis infrastructure that provides continuous, low-overhead summary information on the health and performance of Lustre, as well as on-demand, in-depth problem diagnosis and root-cause analysis. The MELT infrastructure leverages a distributed overlay network to enable monitoring of center-wide Lustre filesystems where clients are located across many network domains. We preview interactive command-line utilities that help administrators and users to observe Lustre performance at various levels of resolution, from individual servers or clients to whole filesystems, including job-level reporting. Finally, we discuss our future plans for automating the root-cause analysis of common Lustre performance problems.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2005

Lessons Learned in the Livingstone 2 on Earth Observing One Flight Experiment

The Livingstone 2 (L2) model-based diagnosis software is a reusable diagnostic tool for monitoring complex systems. In 2004, L2 was integrated with the JPL Autonomous Sciencecraft Experiment (ASE) and deployed on-board Goddard's Earth Observing One (EO-1) remote sensing satellite, to monitor and diagnose the EO-1 space science instruments and imaging sequence. This paper reports on lessons learned from this flight experiment. The goals for this experiment, including validation of minimum success criteria and of a series of diagnostic scenarios, have all been successfully net. Long-term operations in space are on-going, as a test of the maturity of the system, with L2 performance remaining flawless. L2 has demonstrated the ability to track the state of the system during nominal operations, detect simulated abnormalities in operations and isolate failures to their root cause fault. Specific advances demonstrated include diagnosis of ambiguity groups rather than a single fault candidate; hypothesis revision given new sensor evidence about the state of the system; and the capability to check for faults in a dynamic system without having to wait until the system is quiescent. The major benefits of this advanced health management technology are to increase mission duration and reliability through intelligent fault protection, and robust autonomous operations with reduced dependency on supervisory operations from Earth. The work-load for operators will be reduced by telemetry of processed state-of-health information rather than raw data. The long-term vision is that of making diagnosis available to the onboard planner or executive, allowing autonomy software to re-plan in order to work around known component failures. For a system that is expected to evolve substantially over its lifetime, as for the International Space Station, the model-based approach has definite advantages over rule-based expert systems and limit-checking fault protection systems, as these do not scale well. The model-based approach facilitates reuse of the L2 diagnostic software; only the model of the system to be diagnosed and telemetry monitoring software has to be rebuilt for a new system or expanded for a growing system. The hierarchical L2 model supports modularity and expendability, and as such is suitable solution for integrated system health management as envisioned for systems-of-systems.

Hayden, Sandra C.↗

DOE OSTI2022

Using discrete Bayesian networks for diagnosing and isolating cross-level faults in HVAC systems

Fault detection and diagnosis (FDD) technologies are critical to ensure satisfactory building performance, such as reducing energy wastes and negative impacts on occupant comfort and productivity. Existing FDD technologies mainly focus on component-level FDD solutions, which could lead to mis-diagnosis of cross-level faults in heating, ventilating, and air-conditioning (HVAC) systems. Cross-level faults are those faults that occur in one component or subsystem, but cause operational abnormalities in other components or subsystems, and result in a building level performance degradation. How to effectively diagnose the root cause of a cross-level fault is the focus of this study. Here, this paper presents a novel discrete Bayesian Network (DisBN)-based method for diagnosing cross-level faults in an HVAC system commonly used in commercial buildings. A two-level DisBN structure model is developed in this study. The parameters used in the DisBN model are obtained either from expert knowledge or through machine-learning strategies from normal system operation data. Meanwhile, the probability parameters are discretized to incorporate the uncertainties associated with typical expert knowledge. Thus, the developed DisBN method addresses the challenges many other BN based FDD methods face, i.e., the lack of fault data for BN parameter training. The developed DisBN represents causal relationships between a fault and its cross-level system impacts (i.e., fault symptoms or fault indicators) by considering how fault impacts propagate across different levels in an HVAC system. A weather and schedule information-based Pattern Matching (WPM) method is employed to automatically create WPM baseline data sets for each incoming real time snapshot data from the building systems. Consequently, BN inference and real-time diagnostics are achieved by comparing incoming snapshot data and the WPM baseline data set. The proposed method is evaluated using experimental fault data collected in a campus building. Fault diagnosis results demonstrate that the WPM-DisBN method is effective at locating the root causes of cross-level faults in an HVAC system.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2019

System Deterioration Detection and Root Cause Learning on Time Series Graphs

System deterioration detection and root cause analysis is crucial for today's industrial society. However, the design and operation of mechanic system is getting more and more complex, which makes it hard at identifying deterioration with noisy data. Our research focus on solving such problem on time-evolving sensor graphs in a streaming setting. Given a sequence of graphs, the ability to identify 1) any gradual and stable structured change and 2) the root cause components is of importance for early warning and system diagnosis. Existing methods either raise too many false alerts on instant changes or are too sensitive to noise. To address these problems, we propose Robust Failure Detection and Diagnosis (RoFaD). RoFaD can capture failure propagation given a time series of graph. By optimizing a matrix based Taylor expansion, RoFaD can identify system deterioration in the presence of noise and immediate changes, and diagnose the root cause components. Experiments on both synthetic and real world datasets demonstrate that RoFaD is more effective than the popular baselines.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2013

Model-Based Diagnosis and Prognosis of a Water Recycling System

A water recycling system (WRS) deployed at NASA Ames Research Center s Sustainability Base (an energy efficient office building that integrates some novel technologies developed for space applications) will serve as a testbed for long duration testing of next generation spacecraft water recycling systems for future human spaceflight missions. This system cleans graywater (waste water collected from sinks and showers) and recycles it into clean water. Like all engineered systems, the WRS is prone to standard degradation due to regular use, as well as other faults. Diagnostic and prognostic applications will be deployed on the WRS to ensure its safe, efficient, and correct operation. The diagnostic and prognostic results can be used to enable condition-based maintenance to avoid unplanned outages, and perhaps extend the useful life of the WRS. Diagnosis involves detecting when a fault occurs, isolating the root cause of the fault, and identifying the extent of damage. Prognosis involves predicting when the system will reach its end of life irrespective of whether an abnormal condition is present or not. In this paper, first, we develop a physics model of both nominal and faulty system behavior of the WRS. Then, we apply an integrated model-based diagnosis and prognosis framework to the simulation model of the WRS for several different fault scenarios to detect, isolate, and identify faults, and predict the end of life in each fault scenario, and present the experimental results.

Roychoudhury, Indranil↗

DOE OSTI2019

DiagSoftfailure: Automated Soft-Failure Diagnostic Tool Using Machine Learning for Network Users

Network performance problems resulting in slow data communication speed become a significant obstacle for a satisfactory user experience. Currently, there is a lack of an automated diagnostic tool that can help network users to find the complicated network problems that degrade network performance. To fill up this gap, InfoBeyond proposes DiagSoftfailure to provide an automated diagnosis tool. DiagSoftfailure is a machine learning-based approach to infer the location and root cause of network failures that result in performance degradation. As a user-focused tool, it allows a network user to request a diagnosis. DiagSoftfailure can automatically collect the raw packet traces in the network. Base on the network signature, automated classifiers trained by combining supervised and semi-supervised machine learning is used to identify both known and unknown soft-failures in the network. A report allows novice and expert users to understand network problems for troubleshooting. In this document, we describe the DiagSoftfailure diagnosis algorithms and our experiments in a live high-speed network. DiagSoftfailure achieves 96.39% accuracy for known soft-failure diagnosis and 99.82% for unknown soft-failure diagnosis, respectively. Such results demonstrate that DiagSoftfailure is a viable technology. InfoBeyond would develop it as a marketable product that can be used by universities, Labs, DOE, government agencies, organizations, enterprises, and scientific communities to improve their network diagnosis capabilities.

42 ENGINEERING↗