Search NASA⌕ Search

SEARCH · Search NASA

Results for “hardware failure”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Asynchronous distributed-memory task-parallel algorithm for compressible flows on unstructured 3D Eulerian grids

Here, we discuss the implementation of a finite element method, used to numerically solve the Euler equations of compressible flows, using an asynchronous runtime system (RTS). The algorithm is implemented for distributed-memory machines, using stationary unstructured 3D meshes, combining data-, and task-parallelism on top of the Charm++ RTS. Charm++’s execution model is asynchronous by default, allowing arbitrary overlap of computation and communication. Task-parallelism allows scheduling parts of an algorithm independently of, or dependent on, each other. Built-in automatic load balancing enables continuous redistribution of computational load by migration of work units based on real-time CPU load measurement. The RTS also features automatic checkpointing, fault tolerance, resilience against hardware failure, and supports power-, and energy-aware computation. We demonstrate scalability up to 25 x 10 9 cells at $\mathscr{O}$10 4 compute cores and the benefits of automatic load balancing for irregular workloads. The full source code with documentation is available at https://quinoacomputing.org.

42 ENGINEERING↗

Dynamic File Striping and Data Layout Transformation on Parallel System with Fluctuating I/O Workload

As the number of compute cores on modern parallel machines increases to more than hundreds of thousands, scalable and consistent I/O performance is becoming hard to obtain due to fluctuating file system performance. This fluctuation is often caused by rebuilding RAID disk from hardware failures or concurrent jobs competing for I/O. We present a mechanism that stripes across a dynamically-selected subset of I/O servers with the lightest workload to achieve the best I/O bandwidth available from the system. We implement this mechanism into an I/O software layer that enables memory-to-file data layout transformation and allows transparent file partitioning. File partitioning is a technique that divides data among a set of files and manages file access, making data appear as a single file to users. Experimental results on NERSC’s Hopper indicate that our approach effectively isolates I/O variation on shared systems and improves overall I/O performance significantly.

Son, Seung Woo↗

Detecting Anomalies from End-to-End Internet Performance Measurements (PingER) Using Cluster Based Local Outlier Factor

PingER (Ping End-to-End Reporting) is a worldwide end-to-end Internet performance measurement framework. It was developed by the SLAC National Accelerator Laboratory, Stanford, USA and running from the last 20 years. It has more than 700 monitoring agents and remote sites which monitor the performance of Internet links around 170 countries of the world. At present, the size of the compressed PingER data set is about 60 GB comprising of 100,000 flat files. The data is publicly available for valuable Internet performance analyses. However, the data sets suffer from missing values and anomalies due to congestion, bottleneck links, queuing overflow, network software misconfiguration, hardware failure, cable cuts, and social upheavals. Therefore, the objective of this paper is to detect such performance drops or spikes labeled as anomalies or outliers for the PingER data set. In the proposed approach, the raw text files of the data set are transformed into a PingER dimensional model. The missing values are imputed using the k-NN algorithm. The data is partitioned into similar instances using the k-means clustering algorithm. Afterward, clustering is integrated with the Local Outlier Factor (LOF) using the Cluster Based Local Outlier Factor (CBLOF) algorithm to detect the anomalies or outliers from the PingER data. Lastly, anomalies are further analyzed to identify the time frame and location of the hosts generating the major percentage of the anomalies in the PingER data set ranging from 1998 to 2016.

97 MATHEMATICS AND COMPUTING↗

Mapping to Irregular Torus Topologies and Other Techniques for Petascale Biomolecular Simulation, In: SC '14: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Currently deployed petascale supercomputers typically use toroidal network topologies in three or more dimensions. While these networks perform well for topology-agnostic codes on a few thousand nodes, leadership machines with 20,000 nodes require topology awareness to avoid network contention for communication-intensive codes. Topology adaptation is complicated by irregular node allocation shapes and holes due to dedicated input/output nodes or hardware failure. In the context of the popular molecular dynamics program NAMD, we present methods for mapping a periodic 3-D grid of fixed-size spatial decomposition domains to 3-D Cray Gemini and 5-D IBM Blue Gene/Q toroidal networks to enable hundred-million atom full machine simulations, and to similarly partition node allocations into compact domains for smaller simulations using multiple copy algorithms. Additional enabling techniques are discussed and performance is reported for NCSA Blue Waters, ORNL Titan, ANL Mira, TACC Stampede, and NERSC Edison.

Computer Science↗

Fault Oblivious eXascale Whitepaper

In this paper we present a software system which supports dynamic, irregular, adaptive applications. Data objects are created and structured in a hierarchical manner, with replication as needed to provide a high degree of redundancy. The data objects can contain data, code, tasks (work descriptors with references to data, code, and other tasks) and higher level structures such as work queues. The higher level structures benefit from the properties of the data objects: redundant storage to support resiliency in the face of hardware failure; hierarchical structure to optimize use of the HPC system; and a presence of object names, available in the per-user file system name space, which allows any application, not just specially written HPC applications, to make use of the data even while it is on the HPC system. Our use of hierarchy will make the runtime scalable to very large systems. Our use of redundancy will allow programs to be written in a fault-oblivious manner, eliminating the need for system-level checkpointing. Putting data object names into the file system name space allows for interactive use of the system by users. With this approach, we will be able to finally leave the batch era behind, a half-century after the invention of time sharing. We will be able to stop bounding program through- put by the checkpoint interval. Application data will be accessible at any time, not hidden behind opaque 128-bit pointers or MPI ranks, but given a name that is visible everywhere. Programmers can stop laying out data, and thinking about where the data is, and the code is, and the nodes are, and stick with the problem of what the application is supposed to be doing. This work, if it succeeds, will enable scientific computing to scale to the next generation of machines.

99 GENERAL AND MISCELLANEOUS↗

Adding Fault Tolerance to NPB Benchmarks Using ULFM

In the world of high-performance computing, fault tolerance and application resilience are becoming some of the primary concerns because of increasing hardware failures and memory corruptions. While the research community has been investigating various options, from system-level solutions to application-level solutions, standards such as the Message Passing Interface (MPI) are also starting to include such capabilities. The current proposal for MPI fault tolerant is centered around the User-Level Failure Mitigation (ULFM) concept, which provides means for fault detection and recovery of the MPI layer. This approach does not address application-level recovery, which is currently left to application developers. In this work, we present a modification of some of the benchmarks of the NAS parallel benchmark (NPB) to include support of the ULFM capabilities as well as application-level strategies and mechanisms for application-level failure recovery. As such, we present: (i) an application-level library to "checkpoint" and restore data, (ii) extensions of NPB benchmarks for fault tolerance based on different strategies, (iii) a fault injection tool, and (iv) some preliminary results that show the impact of such fault tolerant strategies on the application execution.

97 MATHEMATICS AND COMPUTING↗

HSNMon: High-Speed Monitoring Software for Omni-Path

HSNMon is a robust high speed network monitoring daemon that provides insight into network availability, hardware failure, and network performance for switches and network cards. HSNMon collects counters, forwarding tables, and error information at high resolution using publicly available Omni-Path Management APIs. The collected data can be forwarded into multiple logging infrastructures including syslog and message queuing systems. HSNMon is only the infrastructure for collecting the network data. It does not include any collected data or proprietary LANL information

Holman, Brett↗

Exploring the feasibility of lossy compression for PDE simulations

Checkpoint restart plays an important role in high-performance computing (HPC) applications, allowing simulation runtime to extend beyond a single job allocation and facilitating recovery from hardware failure. Yet, as machines grow in size and in complexity, traditional approaches to checkpoint restart are becoming prohibitive. Current methods store a subset of the application’s state and exploit the memory hierarchy in the machine. However, as the energy cost of data movement continues to dominate, further reductions in checkpoint size are needed. Lossy compression, which can significantly reduce checkpoint sizes, offers a potential to reduce computational cost in checkpoint restart. This article investigates the use of numerical properties of partial differential equation (PDE) simulations, such as bounds on the truncation error, to evaluate the feasibility of using lossy compression in checkpointing PDE simulations. Restart from a checkpoint with lossy compression is considered for a fail-stop error in two time-dependent HPC application codes: PlasComCM and Nek5000. Results show that error in application variables due to a restart from a lossy compressed checkpoint can be masked by the numerical error in the discretization, leading to increased efficiency in checkpoint restart without influencing overall accuracy in the simulation.

97 MATHEMATICS AND COMPUTING↗

Anomaly Detection in Accelerator Facilities Using Machine Learning

Synchrotron light sources are user facilities and usually run about 5000 hours per year to support many beamlines operations in parallel. Reliability is a key parameter to evaluate machine performance. Even many facilities have achieved >95% beam reliability, there are still many hours of unscheduled downtime and every hour lost is a waste of operation costs along with a big impact on individual scheduled user experiments. Preventive maintenance on subsystems and quick recovery from machine trips are the basic strategies to achieve high reliability, which heavily depends on experts’ dedication. Recently, SLAC, APS, and NSLS-II collaborated to develop machine-learning-based approaches aiming to solve both situations, hardware failure prediction and machine failure diagnosis to find the root sources. In this paper, we report our facility operation status, development progress, and plans.

Accelerator Physics↗

Scalable fault tolerance runtime technology for petascale computers (Final Project Report)

With the massive number of components comprising the forthcoming petascale computer systems, hardware failures will be routinely encountered during execution of large-scale applications. Due to the multidisciplinary, multiresolution, and multiscale nature of scientific problems that drive the demand for high end systems, applications place increasingly differing demands on the system resources: disk, network, memory, and CPU. In addition to MPI, future applications are expected to use advanced programming models such as those developed under the DARPA HPCS program as well as existing global address space programming models such as Global Arrays, UPC, and Co-Array Fortran. While there has been a considerable amount of work in fault tolerant MPI with a number of strategies and extensions for fault tolerance proposed, virtually none of advanced models proposed for emerging petascale systems is currently fault aware. To achieve fault tolerance, development of underlying runtime and OS technologies able to scale to petascale level is needed. This project has evaluated range of runtime techniques for fault tolerance for advanced programming models.

97 MATHEMATICS AND COMPUTING↗

2016 High Performance Computing Annual Report

Sandia National Laboratories’ California site is celebrating its 60th anniversary (1956 to 2016), and high performance computing has been a key enabler for its scientists and engineers throughout much of its history. Since its founding, Sandia California has helped pioneer the use of HPC platforms including hosting Sandia’s first Cray-1 supercomputer in the 1970s and supporting development of scalable cluster computing platforms to create a new paradigm for cost-effective supercomputing in the 1990s. Recent decades of investment in creation of scalable application frameworks for scientific computing have also enabled new generations of modeling and simulation codes. These resources have facilitated computational analysis of complex phenomena in diverse applications spanning national defense, energy, and homeland security. Today, Sandia California researchers work with partners in academia, industry, and national labs to evolve the state-of-the-art in HPC, modeling, and data analysis (including foundational capabilities for exascale computing platforms) and apply them in transformational ways. Research efforts include mitigating the effects of silent hardware failures that can jeopardize the results of large-scale computations, developing exascale-capable asynchronous task-parallel programming models and runtime systems, formulating new techniques to better explore and analyze extreme-scale data sets, and increasing our understanding of materials and chemical sciences which has applications spanning nuclear weapons stockpile stewardship to more efficient automobile engines. The following section highlights some of these research and applications projects and further illustrates the breadth of our HPC capabilities.

97 MATHEMATICS AND COMPUTING↗

Vector Calibration for Vertical Tests of Superconducting RF Cavities

We propose procedure for vector calibration of Q 0 vs E acc curve measurement for superconducting resonators. The proposed vector calibration procedure and hardware has several advantages over the techniques currently used for vertical cavity tests. It provides high level of automation and reduces manpower and preparation time for the test; it provides in situ, real time control of hardware parameters and allows to detect hardware failure and operator's errors; it allows to control and exclude the most significant systematic uncertainties, measured values of Q 0 (E acc ) can be made traceable to national standards.

43 PARTICLE ACCELERATORS↗

ECE 4396 (Final Report)

In the summer of 2025, I was fortunate enough intern at Sandia National Laboratories in Albuquerque, New Mexico. I was hired into the Southwest Analysis Laboratories for Semiconductor Advancement (SALSA) intern program. In this internship, I applied my knowledge and skills in electrical engineering to conduct hardware failure analysis. I utilized various failure analyze techniques involving the use of Infrared Thermography (IRT) and Laser Scanning Microscopy (LSM) to test different Application-Specific Integrated Circuits (ASIC) chips that are available in the public market.

42 ENGINEERING↗

Analysis of Real-World Preignition Data Using Neural Networks

Increasing adoption of downsized, boosted, spark-ignition engines has improved vehicle fuel economy, and continued improvement is desirable to reduce carbon emissions in the near-term. However, this strategy is limited by damaging preignition events which can cause hardware failure. Research to date has shed light on various contributing factors related to fuel and lubricant properties as well as calibration strategies, but the causal factors behind an individual preignition cycle remain elusive. If actionable precursors could be identified, mitigation through active control strategies would be possible. This paper uses artificial neural networks to search for identifiable precursors in the cylinder pressure data from a large real-world data set containing many preignition cycles. It is found that while follow-up preignition cycles in clusters can be readily predicted, the initial preignition cycle is not predictable based on features of the cylinder pressure. Further, this indicates that the alternating pattern of preignition cycles within clusters is influenced by the thermodynamic state as reflected in the pressure, but that the trigger for the initial preignition cycle is not thermodynamic in nature, but more likely tied to a critical threshold in the chemistry of the fuel/lubricant mixture in the upper crevice or other factors related to the presence of an ignition source.

33 ADVANCED PROPULSION SYSTEMS↗

Assessing the Impact of Mirror Technology on Driver Perception and Safety: Traditional vs. Camera-Based Systems

Camera-based mirror systems (CBMS) are being adopted by commercial fleets based on the potential improvements to operational efficiency through improved aerodynamics, resulting in better fuel economy, improved maneuverability, and the potential improvement for overall safety. Until CBMS are widely adopted it will be expected that drivers will be required to adapt to both conventional glass mirrors and CBMS which could have potential impact on the safety and performance of the driver when moving between vehicles with and without CBMS. To understand the potential impact to driver perception and safety, along with other human factors related to CBMS, laboratory testing was performed to understand the impact of CBMS and conventional glass mirrors. Drivers were subjected to various, nominal driving scenarios using a truck equipped with conventional glass mirrors, CBMS, and both glass mirrors and CBMS, to observe the differences in metrics such as head and eye movement, reaction time, and perception of distance. The finds from this study will serve as the baseline measurements for future research regarding off-nominal driving scenarios and hardware failures of CBMS, as well as inform potential future policy regarding CBMS for the use in commercial vehicles in lieu of conventional glass mirrors.

Siekmann, Adam [ORNL] (ORCID:0000000284653935)↗

An integrated approach to monitoring the calibration stability of operational dual-polarization radars

The stability of weather radar calibration is a mandatory aspect for quantitative applications, such as rainfall estimation, short-term weather prediction and initialization of numerical atmospheric and hydrological models. Over the years, calibration monitoring techniques based on external sources have been developed, specifically calibration using the Sun and calibration based on ground clutter returns. In this paper, these two techniques are integrated and complemented with a self-consistency procedure and an intercalibration technique. The aim of the integrated approach is to implement a robust method for online monitoring, able to detect significant changes in the radar calibration. The physical consistency of polarimetric radar observables is exploited using the self-consistency approach, based on the expected correspondence between dual-polarization power and phase measurements in rain. This technique allows a reference absolute value to be provided for the radar calibration, from which eventual deviations may be detected using the other procedures. In particular, the ground clutter calibration is implemented on both polarization channels (horizontal and vertical) for each radar scan, allowing the polarimetric variables to be monitored and hardware failures to promptly be recognized. The Sun calibration allows monitoring the calibration and sensitivity of the radar receiver, in addition to the antenna pointing accuracy. It is applied using observations collected during the standard operational scans but requires long integration times (several days) in order to accumulate a sufficient amount of useful data. Finally, an intercalibration technique is developed and performed to compare colocated measurements collected in rain by two radars in overlapping regions. The integrated approach is performed on the C-band weather radar network in northwestern Italy, during July–October 2014. The set of methods considered appears suitable to establish an online tool to monitor the stability of the radar calibration with an accuracy of about 2 dB. In conclusion, this is considered adequate to automatically detect any unexpected change in the radar system requiring further data analysis or on-site measurements.

54 ENVIRONMENTAL SCIENCES↗

Online Diagnostics at Scale

The Oak Ridge Leadership Computing Facility (OLCF) housed at the Oak Ridge National Laboratory recently acquired a 200-cabinet Cray XK6. The computer will primarily provide capability computing cycles to the U.S. Department of Energy (DOE) Office of Science INCITE program. The OLCF has a tradition of installing very large computer systems requiring unique methods in order to achieve production status in the most expeditious and efficient manner. This paper will explore some of the methods that have been used over the years at OLCF to eliminate both early-life hardware failures and ongoing failures giving users a more stable machine for production.

99 GENERAL AND MISCELLANEOUS↗

System for detecting operating errors in a variable valve timing engine using pressure sensors

A method and control module includes a pressure sensor data comparison module that compares measured pressure volume signal segments to ideal pressure volume segments. A valve actuation hardware remedy module performs a hardware remedy in response to comparing the measured pressure volume signal segments to the ideal pressure volume segments when a valve actuation hardware failure is detected.

47 OTHER INSTRUMENTATION↗