Search NASA⌕ Search

SEARCH · Search NASA

Results for “hardware failure”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15

Elimination of Potential Electrical Stress During EMC (CS01) Testing

This viewgraph presentation reviews possible ways to eliminate electrical stress during Electromagneticic Compatibility (EMC) testing. The presentation reviews tests that have had problems due to electrical stress. On December 5, 1995 Cassini Radar instrument failed a functional test in preparation for EMC conducted susceptibility (CSO 1 ) testing. The instrument power supply did not turn on as required, and failure occurred prior to injection of CS test stimulus. A investigation of the failure was conducted. A PSPICE simulation of Cassini Radar 30V line using the EMC test setup was performed; the result of the simulation was an oscillation on the 30V input of the power supply. In another case: on December 28, 1999 an oscillation occurred on the input power line of the SlRTF Infrared Array Camera (IRAC) while preparing to perform CSOI testing, Resulted in damage to flight hardware. Subsequent to failure, JPL provided GSFC history and corrective action from Cassini Radar CSOI test failure GSFC implemented the same corrective action as JPL, except that the value of the resistor connected across the isolation transformer primary winding is 2.5 ohms instead of 50 ohms. Three recommendations are made: (1) Make EMC test community aware of the problem and potential solutions by presenting papers at major environmental test conferences (2) Include warnings and safeguards in EMC test requirements and procedures (3) Try to convince EMC test equipment suppliers to design a CSOl test fixture similar to fixture shown in the diagram

Cassini radar↗

Independent Orbiter Assessment (IOA): Analysis of the DPS subsystem

The results of the Independent Orbiter Assessment (IOA) of the Failure Modes and Effects Analysis/Critical Items List (FMEA/CIL) is presented. The IOA approach features a top-down analysis of the hardware to independently determine failure modes, criticality, and potential critical items. The independent analysis results corresponding to the Orbiter Data Processing System (DPS) hardware are documented. The DPS hardware is required for performing critical functions of data acquisition, data manipulation, data display, and data transfer throughout the Orbiter. Specifically, the DPS hardware consists of the following components: Multiplexer/Demultiplexer (MDM); General Purpose Computer (GPC); Multifunction CRT Display System (MCDS); Data Buses and Data Bus Couplers (DBC); Data Bus Isolation Amplifiers (DBIA); Mass Memory Unit (MMU); and Engine Interface Unit (EIU). The IOA analysis process utilized available DPS hardware drawings and schematics for defining hardware assemblies, components, and hardware items. Each level of hardware was evaluated and analyzed for possible failure modes and effects. Criticality was assigned based upon the severity of the effect for each failure mode. Due to the extensive redundancy built into the DPS the number of critical items are few. Those identified resulted from premature operation and erroneous output of the GPCs.

Lowery, H. J.↗

PRELIMINARY JPSS-3 VIIRS POLARIZATION SENSITIVITY AND COMPARISON WITH S-NPP, JPSS-1 AND -2

The Visible-Infrared Imaging Radiometer Suite (VIIRS) was first launched on-board the Suomi National Polar-orbiting Partnership (S-NPP) spacecraft in October of 2011. There have been three subsequent builds of the VIIRS sensor for the Joint Polar Satellite System (JPSS) program with JPSS-1, -2 and -3 having launch dates of November 2017, March 2022 and 2026 respectively. There is also a JPSS-4 VIIRS, that is in hardware integration during 2020, with a launch date of 2031. VIIRS has 22 bands: 7 thermal emissive bands (TEBs), 14 reflective solar bands (RSBs) and a Day Night Band (DNB). Ocean Color/Chlorophyll (OCC) products use calibrated Science Data Records (SDRs) for bands M1-M7(0.412-0.865μm) to compute their ocean chemistry products. These bands require accurate polarization sensitivity characterization to compensate for polarized upwelling Rayleigh scatter and produce accurate OCC Environment Data Products (EDRs). VIIRS polarization sensitivity requirement failures have driven hardware modifications to the bandpass filters and dichroic beam splitter over the program. This paper will discuss the preliminary JPSS-3 polarization results and how these hardware modifications, as the JPSS program progresses, have affected the sensor performance. Comparisons of the polarization sensitivities between sensor builds will be discussed along with the hardware modifications that contributed to their differences.

VIIRS↗

Preliminary JPSS-3 VIIRS Polarization Sensitivity and Comparison with S-NPP, JPSS-1 and -2

The Visible-Infrared Imaging Radiometer Suite (VIIRS) was first launched on-board the Suomi National Polar-orbiting Partnership (S-NPP) spacecraft in October of 2011. There have been three subsequent builds of the VIIRS sensor for the Joint Polar Satellite System (JPSS) program with JPSS-1, -2 and -3 having launch dates of November 2017, March 2022 and 2026 respectively. There is also a JPSS-4 VIIRS, that is in hardware integration during 2020, with a launch date of 2031. VIIRS has 22 bands: 7 thermal emissive bands (TEBs), 14 reflective solar bands (RSBs) and a Day Night Band (DNB). Ocean Color/Chlorophyll (OCC) products use calibrated Science Data Records (SDRs) for bands M1-M7 (0.412-0.865μm) to compute their ocean chemistry products. These bands require accurate polarization sensitivity characterization to compensate for polarized upwelling Rayleigh scatter and produce accurate OCC Environment Data Products (EDRs). VIIRS polarization sensitivity requirement failures have driven hardware modifications to the bandpass filters and dichroic beam splitter over the program. This paper will discuss the preliminary JPSS-3 polarization results and how these hardware modifications, as the JPSS program progresses, have affected the sensor performance. Comparisons of the polarization sensitivities between sensor builds will be discussed along with the hardware modifications that contributed to their differences.

VIIRS↗

Lox/Gox related failures during Space Shuttle Main Engine development

Specific rocket engine hardware and test facility system failures are described which were caused by high pressure liquid and/or gaseous oxygen reactions. The failures were encountered during the development and testing of the space shuttle main engine. Failure mechanisms are discussed as well as corrective actions taken to prevent or reduce the potential of future failures.

Cataldo, C. E.↗

Lunar Reconnaissance Orbiter (LRO) Sun Safe Mode

The Lunar Reconnaissance Orbiter (LRO), a spacecraft designed and built at the National Aeronautics and Space Administration s (NASA) Goddard Space Flight Center (GSFC) in Greenbelt, MD, was launched on June 18, 2009 from Cape Canaveral. It is currently in orbit about the Moon taking detailed science measurements and providing a highly accurate mapping of the suface in preparation for the future return of astronauts to a permanent moon base. Onboard the spacecraft is a complex set of algorithms designed by the attitude control engineers at GSFC to control the pointig for all operational events, including anomalies that require the spacecraft to be put into a well known attitude configuration for a sufficiently long duration to allow for the investigation and correction of the anomaly. GSFC level requirements state that each spacecraft s control system design must include a configuration for this pointing and lso be able to maintain a thermally safe and power positive attitude. This stable control algorithm for anomalous events is commonly referred to as the safe mode and consists of control logic thatwill put the spacecraft in this safe configuration defined by the spacecraft s hardware, power and environment capabilities and limitations. The LRO Sun Safe mode consists of a coarse sun-pointing set of algorithms that puts the spacecraft into this thermally safe and power positive attitude and can be achieved wihin a required amount of time from any initial attitude, provided that the system momentum is within the momentum capability of the reaction wheels. On LRO the Sun Safe mode makes use of coarse sun sensors (CSS), an inertial reference unit (IRU) and reaction wheels (RW) to slew the spacecraft to a solar inertial pointing. The CSS and reaction wheels have some level of redundancy because of their numbers. However, the IRU is a single-point-failure piece of hardware. Without the rate information provided by the IRU, the Sun Safe control algorithms could not maintain the required pointing, so a sub-mode of the Sun Safe mode that does not use the IRU was designed. This submode, referred to as the Sun Safe Gyroless control mode, consists of an algorithm that estimates rate information from the CSS and the RW measurements. RW momentum information is used to estimate the body rate parallel to the target sunline, which CSS alone would not be able to observe. Sun Safe can be autonomously, or via ground command, entered from any other control mode and in the event the IRU is not providing rate information, the control mode is switched to the gyroless submode. This paper looks at the design of the Sun Safe modes and discusses the constraints placed on the algorithm and how the mode wored around these constraints. Items of particular interest include CSS placement on the Solar Array (SA) and its implications to design, estimation of body rate information for the Sun Safe Gyroless control mode, and the effect of solar eclipse on each of the Sun Safe modes. Placing CSS on the SA was necessary for the means to put the Sun along the targeted sun-line, nominally normal to the SA panels, for all operational considerations. This had design implications for determining a sun vector during normal SA operations, if one or both gimbals become inoperable and when the SA is in a stowed configuration. The ability of body rate estimation in Sun Safe Gyroless not only uses CSS sun vector data but requires RW momentum measuremens to estimate rates parallel to the sun-line. LRO encounters solar eclipses of some length for most of its orbits about the Moon. With the lack of CSS measurement data a design was implemented in both Sun Safe and Sun Safe Gyroless, they differ because of having or not having IRU measurement data, to carry the spacecraft through these eclipse periods. This paper also includes some discussion of sun avoidance and how it affected design decisions during nominal and eclipse perids for each of the Sun Safe modes.

Garrick, Joseph↗

Modeling interconnections of safety and financial performance of nuclear power plants, part 3: Spatiotemporal probabilistic physics-of-failure analysis and its connection to safety and financial performance

Here, this paper is a byproduct of a line of research by the authors to analyze interrelationships of safety and financial performance of nuclear power plants (NPPs). The result of this line of research is summarized in three parts: Part 1 covers a categorical review of relevant literature and the theoretical bases that support the methodological developments in Part 2. Part 2 introduces an Integrated Enterprise Risk Management (I-ERM) methodological framework to quantify the interconnections of safety and financial performance with a focus on operation and maintenance (O&M) of NPPs. Part 2 has also demonstrated the applicability and values of the I-ERM methodology through an NPP case study. This paper is Part 3, where detailed development and implementation of one of the I-ERM modules, i.e., probabilistic physics-of-failure (PPoF) analysis, and its connection with safety and financial performance is reported. In this article, the physical failure modeling for hardware components is advanced by incorporating finite element analysis (FEA) into PPoF analysis and coupling the FEA-based PPoF with the maintenance performance through a renewal process model. This article covers two scientific contributions: (i) first-of-its-kind incorporation of FEA into the PPoF model of thermal fatigue for NPP components; and (ii) advancing the interface between the PPoF analysis and the renewal process model in order to deal with spatiotemporal FEA outputs and to efficiently estimate the physical transition rates even when the PPoF outputs are dominated by success data. Through the incorporation of FEA, the resolution of the PPoF analysis is enhanced as spatiotemporal conditions such as stress and temperature can be considered explicitly instead of relying on simplified assumptions or analytical models with reduced spatiotemporal dimensions. To demonstrate an application of the FEA-based PPoF analysis and its coupling with maintenance through the renewal process model, a case study is conducted using excess letdown elbow piping in the chemical and volume control system of a Pressurized Water Reactor.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DAQExpert the service to increase CMS data-taking efficiency

The Data Acquisition (DAQ) system of the Compact Muon Solenoid (CMS) experiment at the LHC is a complex system responsible for the data readout, event building and recording of accepted events. Its proper functioning plays a critical role in the data-taking efficiency of the CMS experiment. In order to ensure high availability and recover promptly in the event of hardware or software failure of the subsystems, an expert system, the DAQ Expert, has been developed. It aims at improving the data taking efficiency, reducing the human error in the operations and minimising the on-call expert demand. Introduced in the beginning of 2017, it assists the shift crew and the system experts in recovering from operational faults, streamlining the post mortem analysis and, at the end of Run 2, triggering fully automatic recovery without human intervention. DAQ Expert analyses the real-time monitoring data originating from the DAQ components and the high-level trigger updated every few seconds. It pinpoints data flow problems, and recovers them automatically or after given operator approval. We analyse the CMS downtime in the 2018 run focusing on what was improved with the introduction of automated recovery; present challenges and design of encoding the expert knowledge into automated recovery jobs. Furthermore, we demonstrate the web-based, ReactJS interfaces that ensure an effective cooperation between the human operators in the control room and the automated recovery system. We report on the operational experience with automated recovery.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Analyzing a Five-Year Failure Record of a Leadership-Class Supercomputer

Extreme-scale computing systems are required to solve some of the grand challenges in science and technology. From astrophysics to molecular biology, supercomputers are an essential tool to accelerate scientific discovery. However, large computing systems are prone to failures due to their complexity. It is crucial to develop an understanding of how these systems fail to design reliable supercomputing platforms for the future. This paper examines a five-year failure and workload record of a leadership-class supercomputer. To the best of our knowledge, five years represents the vast majority of the lifespan of a supercomputer. This is the first time such analysis is performed on a top 10 modern supercomputer. We performed a failure categorization and found out that: i) most errors are GPU-related, with roughly 37% of them being double-bit errors on the cards; ii) failures are not evenly spread across the physical machine, with room temperature presumably playing a major role; and iii) software errors of the system bring down several nodes concurrently. Our failure rate analysis unveils that: i) the system consistently degrades, being at least twice as reliable at the beginning, compared to the end of the period; ii) Weibull distribution closely fits the mean-time-between-failure data; and iii) hardware and software errors show a markedly different pattern. Finally, we correlated failure and workload records to reveal that: i) failure and workload records are weakly correlated, except for certain types of failures when segmented by the hours of the day; ii) several categories of failures make jobs crash within the first minutes of execution; and iii) a significant fraction of failed jobs exhaust the requested time with a disregard of when the failure occurred during execution.

Rojas, Elvis↗

MO-E-9A-01: Risk Based Quality Management: TG100 In Action

One of the goals of quality management in radiation therapy is to gain high confidence that patients will receive the prescribed treatment correctly. To accomplish these goals professional societies such as the American Association of Physicists in Medicine (AAPM) has published many quality assurance (QA), quality control (QC), and quality management (QM) guidance documents. In general, the recommendations provided in these documents have emphasized on performing device-specific QA at the expense of process flow and protection of the patient against catastrophic errors. Analyses of radiation therapy incidents find that they are most often caused by flaws in the overall therapy process, from initial consult through final treatment, than by isolated hardware or computer failures detectable by traditional physics QA. This challenge is shared by many intrinsically hazardous industries. Risk assessment tools and analysis techniques have been developed to define, identify, and eliminate known and/or potential failures, problems, or errors, from a system, process and/or service before they reach the customer. These include, but are not limited to, process mapping, failure modes and effects analysis (FMEA), fault tree analysis (FTA), and establishment of a quality management program that best avoids the faults and risks that have been identified in the overall process. These tools can be easily adapted to radiation therapy practices because of their simplicity and effectiveness to provide efficient ways to enhance the safety and quality of treatment processes. Task group 100 (TG100) of AAPM has developed a risk-based quality management program that uses these tools. This session will be devoted to a discussion of these tools and how these tools can be used in a given radiotherapy clinic to develop a risk based QM program. Learning Objectives: Learn how to design a process map for a radiotherapy process. Learn how to perform a FMEA analysis for a given process. Learn what Fault tree analysis is all about. Learn how to design a quality management program based upon the information obtained from process mapping, FMEA and FTA.

60 APPLIED LIFE SCIENCES↗

Compressed Sensing with Sparse Corruptions: Fault-Tolerant Sparse Collocation Approximations

The recovery of approximately sparse or compressible coefficients in a polynomial chaos expansion is a common goal in many modern parametric uncertainty quantification (UQ) problems. However, relatively little effort in UQ has been directed toward theoretical and computational strategies for addressing the sparse corruptions problem, where a small number of measurements are highly corrupted. Such a situation has become pertinent today since modern computational frameworks are sufficiently complex with many interdependent components that may introduce hardware and software failures, some of which can be difficult to detect and result in a highly polluted simulation result. In this paper we present a novel compressive sampling--based theoretical analysis for a regularized $\ell^1$ minimization algorithm that aims to recover sparse expansion coefficients in the presence of measurement corruptions. Our recovery results are uniform (the theoretical guarantees hold for all compressible signals and compressible corruptions vectors) and prescribe algorithmic regularization parameters in terms of a user-defined a priori estimate on the ratio of measurements that are believed to be corrupted. We also propose an iteratively reweighted optimization algorithm that automatically refines the value of the regularization parameter and empirically produces superior results. Lastly, our numerical results test our framework on several medium to high dimensional examples of solutions to parameterized differential equations and demonstrate the effectiveness of our approach.

97 MATHEMATICS AND COMPUTING↗

Compressed sensing with sparse corruptions: Fault-tolerant sparse collocation approximations

The recovery of approximately sparse or compressible coefficients in a polynomial chaos expansion is a common goal in many modern parametric uncertainty quantification (UQ) problems. However, relatively little effort in UQ has been directed toward theoretical and computational strategies for addressing the sparse corruptions problem, where a small number of measurements are highly corrupted. Such a situation has become pertinent today since modern computational frameworks are sufficiently complex with many interdependent components that may introduce hardware and software failures, some of which can be difficult to detect and result in a highly polluted simulation result. In this paper we present a novel compressive sampling-based theoretical analysis for a regularized t1 minimization algorithm that aims to recover sparse expansion coefficients in the presence of measurement corruptions. Our recovery results are uniform (the theoretical guarantees hold for all compressible signals and compressible corruptions vectors), and prescribe algorithmic regularization parameters in terms of a user-defined a priori estimate on the ratio of measurements that are believed to be corrupted. We also propose an iteratively reweighted optimization algorithm that automatically refines the value of the regularization parameter, and empirically produces superior results. Our numerical results test our framework on several medium-to-high dimensional examples of solutions to parameterized differential equations, and demonstrate the effectiveness of our approach.

97 MATHEMATICS AND COMPUTING↗

Report on local data recovery approaches suitable for weather and climate prediction (Deliverable 1.3) (V.1.0)

Numerical weather and climate prediction rates as one of the scientific applications whose accuracy improvements greatly depend on the growth of the available computing power. As the number of cores in top computing facilities pushes into the millions, increasing average frequency of hardware and software failures forces users to review their algorithms and systems in order to protect simulations from breakdown. This report surveys approaches for fault-tolerance in numerical algorithms and system resilience in parallel simulations from the perspective of numerical weather and climate prediction systems. A selection of existing strategies is analyzed, featuring interpolation-restart and compressed checkpointing for the numerics, in-memory checkpointing, user-level failure mitigation-based and backup-based methods for the systems. Numerical examples showcase the performance of the techniques in addressing faults, with particular emphasis on iterative solvers for linear systems, a staple of atmospheric fluid flow solvers. The potential impact of these strategies is discussed in relation to current development of numerical weather prediction algorithms and systems towards the exascale. Trade-offs between performance, efficiency and effectiveness of resiliency strategies are analyzed and some recommendations outlined for future developments.

97 MATHEMATICS AND COMPUTING↗

Extension of 4-8 Texture Hierarchies to Large Video Processing and Visualization

The purpose of this Techbase was to reduce to practice the tiled 4-8 texture hierarchy for the display of video imagery (i.e. sequences of frames). The immediate intent was to demonstrate its use in the analysis and display of sensor imagery. As sensors are increasing in resolution the physical amount of imagery that needs to be displayed can quickly overwhelm most display systems. For example, a sensor with a horizontal resolution of over 8000 pixels would generate an image over 10 feet wide on a standard 72 DPI display. Breaking an image into tiles, and then decomposing each tile into a multiresolution hierarchy, allows a user (or software) to efficiently select and display only those parts of the image that are of interest to the user. The originator of the idea of 4-8 Texture Hierarchies was Dr. Mark Duchaineau, and we consulted with him in much of our work. We also consulted with Dan Knight, from SequoiaTek Corp., who is a contractor responsible for implementing the viewers for our applications. Most of the code for actual 4-8 Texture Hierarchy generation already existed; a large focus of the Techbase was to determine how to best use what was available for video imagery. The majority of progress was made in specifying and implementing the software framework, which turned out to be rather involved. This framework is to support the creation, storage, and display of images, both tiled and untiled. A first albeit incomplete version was successfully tested in the field in August 2007. The framework structures the process of collecting and processing images conceptually as a pipeline, where work is passed along and a different operation is performed at each stage. In practice, the pipeline is implemented by a group of processes (not threads), or 'workers', each responsible for a specific type of operation. Associated with these workers is a pool of memory (cache). As each process finishes its work, it places the results into the cache and sends a message to the process responsible for the next operation, including in the message an identifier allowing the recipient to locate the data to be worked on. Conceptually this is akin to transferring mail from person to person by placing it into and removing it from a group of mail slots, rather than handing it to the next person directly. So for example, to capture, tile, and store imagery to disk, one worker would be responsible for obtaining an image from an electro-optical sensor. Another would be responsible for breaking the image into tiles, and another for performing a decorrelating transform for entropy reduction and multiresolution representation. Still another would be responsible for compression, and another for storage. To maximize robustness in an experimental environment, we created a manager that could monitor, kill, restart, and configure the entire processing pipeline, including the shared cache and worker processes. In the event of hardware or software failure, individual elements of the pipeline were restored automatically and with minimum data loss. Working with Mark Duchaineau we attempted to determine a set of ideal coefficients for the wavelet used to create the multiresolution hierarchy. While a workable set exists, it produces undesirable artifacts at high levels of loss, so more work needs to be done to identify a more ideal set. The tiler and data structures for tiling and display have been added to the code base we developed. As currently implemented it takes approximately one second to tile and build a multiresolution hierarchy for an 11-megapixel image on a 3.8 GHz Pentium 4. This is too slow for some uses, so some work will probably need to be done to optimize the code. We may be able to leverage commodity graphics processors to gain extra speed. With the advent of quad-core CPUs it may also be possible to get required throughput by tiling the image and then having several processes run in parallel to build the multiresolution hierarchies for the resulting tiles. Additionally, more work needs to be done to define a data format for storing image tiles on disk, as well as retrieving and displaying them. As this functionality is desirable (and required, if we are to make large data easily viewable) we will be spending a lot of time developing this. As a final note, it was recently brought to our attention that Oracle provides an extension to its database technology that provides most, if not all, of the functionality that we are working on as part of this project. We will be investigating this further; if a suitable solution already exists it may be more valuable time-wise to integrate it into projects that require storing and displaying large images.

99 GENERAL AND MISCELLANEOUS↗

A specification-based approach to concurrent structure verification in multiprocessor systems

A recently initiated research project concerned with the concurrent detection of software errors and errors due to physical failures in the hardware of multiprocessor systems is described in this paper. An approach to error detection is described, which is specification based and relies on the structural verification of program control flow and data structure integrity. The techniques discussed utilize the hardware redundancy inherent in parallel processing systems to provide verification of both program structure and data concurrently with program execution.

Fuchs, W. Kent↗