SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
- Objective - Background - Capability Resource Tables (CRT) - Root Cause Analysis(RCA) - Error types - RCA - Fishbone; 5 whys - Lessons Learned
The primary salt pump installed in the high-temperature FLUoride Salt Test Facility (FLUSTFA) was successfully operated for some time, but later ceased operation. To understand what occurred, a Root Cause Analysis (RCA) was performed. Steps taken to try to get the pump operational include adjusting the shaft position, increasing the heating power of the tape heaters on the pump volute, and manually rotating the pump shaft. While removing the insulation, corrosion was noted on the outside of the pump volute, and decolorization of the insulation and tape heaters was observed. Significant corrosion products were also observed in the pump itself and the piping connected to the pump. The nitrogen cover gas was maintained from before salt was introduced into the loop until the pump was dismounted and continues to be maintained even after the pump was removed. After considering probable scenarios, causes were assigned and corrective actions were developed to prevent those causes. Then, the RCA was presented to an advisory committee for review, the “Review Committee,” consisting of experts in large molten salt systems: Brandon Haugh, David Holcomb, Kevin Robb, and Vicente Rojas. As a result, the advisory committee provided comprehensive feedback, which have been incorporated into a revised RCA. Findings have then been summarized and reported in this publication.
Explore the source record for details and available documents.
The medical system for spaceflight thus far has been designed to support missions in low earth orbit (LEO). Crew capabilities are limited and heavily dependent on the team of medical support staff at Mission Control Center (MCC) to guide diagnosis and management. However, missions to the Moon and Mars will suffer from several constraints that will make this ground support focused approach to care ineffective. In order to update and modify medical system design, NASA has relied on Probabilistic Risk Assessment (PRA) modeling to mitigate medical risk through trade space analysis. Specifically, capability resource tables (CRT’s) were developed to create a dataset of resources required to manage a list of accepted medical conditions significant in exploration spaceflight. With 120 conditions, this dataset contained hundreds of capabilities and thousands of resources with tens of thousands of cells of data. Initially these tables were built in excel for high throughput during development, but ultimately had to be transferred, managed, and modified into the Evidence Library database for modeling purposes. The process of collating and reviewing the Evidence Library revealed numerous errors in the dataset that had to be corrected through iterative changes. Several error types emerged during this process and can be broken into specific classifications defined as “input”, “transcription”, “structural”, “branching”, and “information”. In reviewing these error types through the root cause analysis (RCA) approach, we were able to identify the contributors to these errors which included single data review points, changing product end goals, limited software selection, time constraints and several others. By reviewing and evaluating the underlying causes we can provide possible system improvements that can be implemented for current and future data management in PRA model inputs.
Abstract Protonic ceramic electrochemical cells (PCECs) offer promising paths for energy storage and conversion. Despite considerable achievements made, PCECs still face challenges such as physiochemical compatibility between componenets and suboptimal solid–solid contact at the interfaces between the electrolytes and electrodes. In this study, a novel approach is proposed that combines in situ electrochemical characterization of interfacial electrical sensor embedded PCECs and machine learning to quantify the contributions of different cell components to total degradation, as well as to predict the remaining useful life. The experimental results suggest that the overpotential induced by the oxygen electrode is 48% less than that of oxygen electrode/electrolyte interfacial contact for up to 1171 h. The data‐driven machine learning simulation predicts the RUL of up to 2132 h. The root cause of degradation is overpotential increase induced by oxygen electrode, which accounts for 82.9% of total cell degradation. The success of the failure diagnostic model is demonstrated by its consistency with degradation modes that do not manifest in electrolysis fade during early real operations. This synergistic approach provides valuable insights into practical failure diagnosis of PCECs and has the potential to revolutionize their development by enabling improved performance prediction and material selection for enhanced durability and efficiency.
On July 7 2020, approximately 1,000 MW reduction in solar photovoltaic (PV) output was experienced by the bulk power system in Southern California after a three-phase fault. Disturbance analysis indicates that instantaneous AC current caused inverter tripping. The projection of the root cause of AC overcurrent from the NERC report is that inverter current control is not tight. This letter provides a quantitative analysis on ac overcurrent and addresses the following questions: (i) what type of current controls may lead to ac overcurrent; and (ii) how controller parameters and grid strength influence ac overcurrent. Here, the letter provides frequency-domain analysis and time-domain simulation results of a grid-integrated inverter to illustrate the effect of control parameters and grid strength on ac overcurrent.
The NASA's Evolutionary Xenon Thruster (NEXT) project is developing an advanced ion propulsion system for future NASA missions for solar system exploration. A critical element of the propulsion system is the Power Processing Unit (PPU) which supplies regulated power to the key components of the thruster. The PPU contains six different power supplies including the beam, discharge, discharge heater, neutralizer, neutralizer heater, and accelerator supplies. The beam supply is the largest and processes up to 93+% of the power. The NEXT PPU had been operated for approximately 200+ hours and has experienced a series of three capacitor failures in the beam supply. The capacitors are in the same, nominally non-critical location the input filter capacitor to a full wave switching inverter. The three failures occurred after about 20, 30, and 135 hours of operation. This paper provides background on the NEXT PPU and the capacitor failures. It discusses the failure investigation approach, the beam supply power switching topology and its operating modes, capacitor characteristics and circuit testing. Finally, it identifies root cause of the failures to be the unusual confluence of circuit switching frequency, the physical layout of the power circuits, and the characteristics of the capacitor.
The NASA s Evolutionary Xenon Thruster (NEXT) project is developing an advanced ion propulsion system for future NASA missions for solar system exploration. A critical element of the propulsion system is the Power Processing Unit (PPU) which supplies regulated power to the key components of the thruster. The PPU contains six different power supplies including the beam, discharge, discharge heater, neutralizer, neutralizer heater, and accelerator supplies. The beam supply is the largest and processes up to 93+% of the power. The NEXT PPU had been operated for approximately 200+ hr and has experienced a series of three capacitor failures in the beam supply. The capacitors are in the same, nominally non-critical location-the input filter capacitor to a full wave switching inverter. The three failures occurred after about 20, 30, and 135 hr of operation. This paper provides background on the NEXT PPU and the capacitor failures. It discusses the failure investigation approach, the beam supply power switching topology and its operating modes, capacitor characteristics and circuit testing. Finally, it identifies root cause of the failures to be the unusual confluence of circuit switching frequency, the physical layout of the power circuits, and the characteristics of the capacitor.
Dynamic phenomena linked to inverter-based resources (IBRs) have gained global attention. Several IBR-induced dynamics have caused bulk power system-connected wind or solar power plants to trip, and some have even led to widespread outages. In addition, many oscillations have been observed involving IBR power plants. In 2023, the IEEE Power & Energy Society (PES) IBR Subsynchronous Oscillations (SSO) task force published a journal article, “Real-World Subsynchronous Oscillation Events in Power Grids With High Penetrations of Inverter-Based Resources,” in which 19 IBR oscillation events were examined for their causation. Earlier in 2020, another PES task force article, “Definition and Classification of Power System Stability-Revisited & Extended,” authored by prominent academics, introduced converter-driven stability as a new category of stability. The international power grid industry community also took action by publishing the CIGRE Green Book, Power System Dynamic Modelling and Analysis in Evolving Networks (led by Babak Badrzadeh and Zia Emin) in 2024. In August 2024, the Energy Systems Integration Group (ESIG) released a practical guide led by Nick Miller, “Diagnosis and Mitigation of Observed Oscillations in IBR-Dominant Power System: A Practical Guide.” The goal of the guide is to assist practicing engineers in making initial judgments and conducting detailed analyses about oscillations. Finally, when addressing the classification of stability and oscillations, the guide emphasizes a causality-based taxonomy for grouping, such as voltage control-induced oscillations, synchronization-induced oscillations, and frequency or active power control-induced oscillations.
A gradual, but persistent, decrease in the optical throughput was detected during the early commissioning phase for the Suomi National Polar-Orbiting Partnership (SNPP) Visible Infrared Imager Radiometer Suite (VIIRS) Near Infrared (NIR) bands. Its initial rate and unknown cause were coincidently coupled with a decrease in sensitivity in the same spectral wavelength of the Solar Diffuser Stability Monitor (SDSM) raising concerns about contamination or the possibility of a system-level satellite problem. An anomaly team was formed to investigate and provide recommendations before commissioning could resume. With few hard facts in hand, there was much speculation about possible causes and consequences of the degradation. Two different causes were determined as will be explained in this paper. This paper will describe the build and test history of VIIRS, why there were no indicators, even with hindsight, of an on-orbit problem, the appearance of the on-orbit anomaly, the initial work attempting to understand and determine the cause, the discovery of the root cause and what Test-As-You-Fly (TAYF) activities, can be done in the future to greatly reduce the likelihood of similar optical anomalies. These TAYF activities are captured in the lessons learned section of this paper.
The complex interdependencies of cyber systems (sensors and communications), physical grids and associated electricity market operations make protecting electric power grids a significant challenge. The energy sector is constantly under new, targeted, advanced and dangerous cyber-attacks that have the potential to result in the loss of human life. These threats are further exacerbated by our need to modernize the grid. One focus of cyber security research in smart grids is the securing of the SCADA system through advanced intrusion detection systems (IDS) and bad data detection algorithms in state estimation. These methods either require full knowledge of the system topology and parameters or fail to understand the physical behaviors under attack. WISP (Watching grid Infrastructure Stealthily through Proxies) is designed to provide additional protection to the power grid using only publicly available data. In particular, WISP exploits the spatio-temporal nature of the real time locational marginal prices (LMPs), in conjunction with other information such as bids, weather, outages and load data to analyze anomalous power pricing behaviors and then correlate those observations to localize regions of interest and identify potential cyber events. WISP is non-intrusive as the tool is deployed as a service in the Cloud or on premise and provides reliable information to system operators for enhanced situational awareness, without impeding energy delivery functions. The WISP technology comprises three modules: the data-driven anomaly detection core, the vulnerability and risk analysis and the root cause analysis. The data-driven anomaly detection core performs the tasks of feature selection, anomaly detection and attack region localization. The vulnerability and risk analysis module provides system level information of the vulnerable variables and times, assisting the operators in selecting monitoring and protection nodes. The root cause analysis module takes the detection results and identifies potential operational conditions that contribute to the detected anomalies. In Phase I, we have demonstrated the feasibility and effectiveness of WISP. We developed a realistic electricity market simulator capable of generating normal and attack market data under various operational conditions. We developed a series of cyber-attack detection and analysis algorithms and evaluated them under multiple data sources. Finally, we integrated all modules into an end-to-end software, providing functions for data management, data analytics and visualization. Specifically, we have achieved: (i) real-time data acceptance from external utility interfaces with >99% acceptance rate; (ii) high performance anomaly detection algorithms with >98% detection accuracy and <0.1% false alarm rate; and (iii) ultra-low computing delay <50 milliseconds. Additionally, our team developed algorithms to identify the vulnerable variables in electricity market operations and root cause analysis functions to identify major contributors to the price spikes. These ancillary modules are necessary when deploying WISP in real world industry environment. In Phase II, we have demonstrated the effectiveness of WISP software on realistic largescale power systems. We performed red team testing for the Phase I WISP software and identified software vulnerabilities and implemented corresponding mitigation solutions. We adapted the electricity market simulator for the Texas synthetic 2000-bus system and generated datasets for the false data injection attacks. We created database and visualization interfaces for the Texas system and the ISO New England system. We performed software optimization in terms of operation efficiency, computing speed and detection accuracy. Finally, we tested the software on the Texas system and the ISO New England system and evaluated the detection performance. Overall, we achieved above 89% detection rate, below 3% false alarm rate and below 37 seconds of end-to-end detection delay.
Electronic logbooks contain valuable information about activities and events concerning their associated particle accelerator facilities. However, the highly technical nature of logbook entries can hinder their usability and automation. As natural language processing (NLP) continues advancing, it offers opportunities to address various challenges that logbooks present. This work explores jointly testing a tailored Retrieval Augmented Generation (RAG) model for enhancing the usability of particle accelerator logbooks at institutes like DESY, BESSY, Fermilab, BNL, SLAC, LBNL and CERN. The RAG model uses a corpus built on logbook contributions and aims to unlock insights from these logbooks by leveraging retrieval over facility datasets, including discussion about potential multimodal sources. Our goals are to increase the FAIR-ness (findability, accessibility, interoperability, and reusability) of logbooks by exploiting their information content to streamline everyday use, to enable macro-analysis for root cause analysis, and to facilitate problem-solving automation.
Electronic logbooks contain valuable information about activities and events concerning their associated particle accelerator facilities. However, the highly technical nature of logbook entries can hinder their usability and automation. As natural language processing (NLP) continues advancing, it offers opportunities to address various challenges that logbooks present. This work explores jointly testing a tailored Retrieval Augmented Generation (RAG) model for enhancing the usability of particle accelerator logbooks at institutes like DESY, BESSY, Fermilab, BNL, SLAC, LBNL, and CERN. The RAG model uses a corpus built on logbook contributions and aims to unlock insights from these logbooks by leveraging retrieval over facility datasets, including discussion about potential multimodal sources. Our goals are to increase the FAIR-ness (findability, accessibility, interoperability, and reusability) of logbooks by exploiting their information content to streamline everyday use, enable macro-analysis for root cause analysis, and facilitate problem-solving automation.
Systems theoretic process analysis (STPA) is becoming an increasingly popular technique to assess how complex digital software systems can fail. Rather than defining failures by their observable failure events, which may be sparse especially for safety rated nuclear digital instrumentation and control systems (DI&C), failures are defined as postulated unsafe actions under specific contextual conditions. This permits a top-down analysis of system hazards and identifies whether imposed constraints and requirements can sufficiently address undesirable hazards. However, STPA is a qualitative approach at identifying inadequacies in the development process and cannot currently be used to quantify unsafe action likelihoods for probabilistic risk assessment. Therefore, in this work, we examine the root causes of software failure and explore whether a consistent correlation can be linked to specific unsafe action classes. We implement Lbl2Vec, an unsupervised document classification and retrieval algorithm, on a database of 4,096 software defect reports acquired from various open-source software systems. By analyzing sentence structure, embedded labels, and word vectors, we show that certain defect types positively correlate to specific unsafe action classes over others. The correlations developed can be used to estimate the failure probability of safety intended DI&C systems which provides a licensing basis for nuclear plant modernization efforts.
Systems theoretic process analysis (STPA) is becoming an increasingly popular technique to assess how complex digital software systems can fail. Rather than defining failures by their observable failure events, which may be sparse especially for safety rated nuclear digital instrumentation and control systems (DI&C), failures are defined as postulated unsafe actions under specific contextual conditions. This permits a top-down analysis of system hazards and identifies whether imposed constraints and requirements can sufficiently address undesirable hazards. However, STPA is a qualitative approach at identifying inadequacies in the development process and cannot currently be used to quantify unsafe action likelihoods for probabilistic risk assessment. Therefore, in this work, we examine the root causes of software failure and explore whether a consistent correlation can be linked to specific unsafe action classes. We implement Lbl2Vec, an unsupervised document classification and retrieval algorithm, on a database of 4,096 software defect reports acquired from various open-source software systems. By analyzing sentence structure, embedded labels, and word vectors, we show that certain defect types positively correlate to specific unsafe action classes over others. The correlations developed can be used to estimate the failure probability of safety intended DI&C systems which provides a licensing basis for nuclear plant modernization efforts.
This paper discusses the performance analysis effort being carried out in the NASA Deep Space Network. The activity involves root cause analysis of failures and assessment of key performance metrics. The root cause analysis helps pinpoint the true cause of observed problems so that proper correction can be effected. The assessment currently focuses on three aspects: (1) data delivery metrics such as Quantity, Quality, Continuity, and Latency; (2) link-performance metrics such as antenna pointing, system noise temperature, Doppler noise, frequency and time synchronization, wide-area-network loading, link-configuration setup time; and (3) reliability, maintainability, availability metrics. The analysis establishes whether the current system is meeting its specifications and if so, how much margin is available. The findings help identify the weak points in the system and direct attention of programmatic investment for performance improvement.
This two day workshop will discuss a range of topics, including root cause analysis, physics-of-failure principles and failure mechanisms in printed circuit boards. Printed circuit boards (PCBs) are the baseline for electronics manufacturing upon which electronic components are mounted and formed into electronic systems. PCBs are used in a variety of electronic circuits from simple one-transistor amplifiers to large super computers. A PCB serves three main functions: 1) it provides the necessary mechanical support for the components in the circuit 2) it provides the necessary electrical interconnections, and 3) it bears some form of legend which identifies the components it carries. The failure modes on the PCBs can be categorized in a hierarchical structure, in which the mechanisms and causes are site or location dependant. Specimen preparation techniques, non-destructive and destructive analysis, and materials characterization will also be discussed. The first day of the workshop will present methodologies for identifying potential failure mechanisms in electronics based on the failure history and, systematic approaches to root cause analysis. The second day will cover failure analysis techniques geared towards various failure mechanisms, along with numerous component and PCB assembly failure analysis case studies that illustrate the techniques and analysis. Failure analysis case studies will be used to illustrate the techniques and analysis principles to arrive at the root cause(s) of field failures on printed circuit boards, active components, and assemblies.