Search NASA⌕ Search

SEARCH · Search NASA

Results for “checkpointing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Space Reclamation for Uncoordinated Checkpointing in Message-Passing Systems

Checkpointing and rollback recovery are techniques that can provide efficient recovery from transient process failures. In a message-passing system, the rollback of a message sender may cause the rollback of the corresponding receiver, and the system needs to roll back to a consistent set of checkpoints called recovery line. If the processes are allowed to take uncoordinated checkpoints, the above rollback propagation may result in the domino effect which prevents recovery line progression. Traditionally, only obsolete checkpoints before the global recovery line can be discarded, and the necessary and sufficient condition for identifying all garbage checkpoints has remained an open problem. A necessary and sufficient condition for achieving optimal garbage collection is derived and it is proved that the number of useful checkpoints is bounded by N(N+1)/2, where N is the number of processes. The approach is based on the maximum-sized antichain model of consistent global checkpoints and the technique of recovery line transformation and decomposition. It is also shown that, for systems requiring message logging to record in-transit messages, the same approach can be used to achieve optimal message log reclamation. As a final topic, a unifying framework is described by considering checkpoint coordination and exploiting piecewise determinism as mechanisms for bounding rollback propagation, and the applicability of the optimal garbage collection algorithm to domino-free recovery protocols is demonstrated.

Wang, Yi-Min↗

Hybrid Checkpointing for MPI Jobs in HPC Environments

As the core count in high-performance computing systems keeps increasing, faults are becoming common place. Check pointing addresses such faults but captures full process images even though only a subset of the process image changes between checkpoints. We have designed a hybrid check pointing technique for MPI tasks of high-performance applications. This technique alternates between full and incremental checkpoints: At incremental checkpoints, only data changed since the last checkpoint is captured. Our implementation integrates new BLCR and LAM/MPI features that complement traditional full checkpoints. This results in significantly reduced checkpoint sizes and overheads with only moderate increases in restart overhead. After accounting for cost and savings, benefits due to incremental checkpoints are an order of magnitude larger than overheads on restarts. We further derive qualitative results indicating an optimal balance between full/incremental checkpoints of our novel approach at a ratio of 1:9, which outperforms both always-full and always-incremental check pointing.

Wang, Chao↗

Efficient Checkpointing of Virtual Machines using Virtual Machine Introspection

Cloud Computing environments rely heavily on system-level virtualization. This is due to the inherent benefits of virtualization including fault tolerance through checkpoint/restart (C/R) mechanisms. Because clouds are the abstraction of large data centers and large data centers have a higher potential for failure, it is imperative that a C/R mechanism for such an environment provide minimal latency as well as a small checkpoint file size. Recently, there has been much research into C/R with respect to virtual machines (VM) providing excellent solutions to reduce either checkpoint latency or checkpoint file size. However, these approaches do not provide both. This paper presents a method of checkpointing VMs by utilizing virtual machine introspection (VMI). Through the usage of VMI, we are able to determine which pages of memory within the guest are used or free and are better able to reduce the amount of pages written to disk during a checkpoint. We have validated this work by using various benchmarks to measure the latency along with the checkpoint size. With respect to checkpoint file size, our approach results in file sizes within 24% or less of the actual used memory within the guest. Additionally, the checkpoint latency of our approach is up to 52% faster than KVM s default method.

Aderholdt, Ferrol↗

The Effects of G2-Phase Enrichment and Checkpoint Abrogation on Low-Dose Hyper-Radiosensitivity

Purpose: An association between low-dose hyper-radiosensitivity (HRS) and the 'early' G2/M checkpoint has been established. An improved molecular understanding of the temporal dynamics of this relationship is needed before clinical translation can be considered. This study was conducted to characterize the dose response of the early G2/M checkpoint and then determine whether low-dose radiation sensitivity could be increased by synchronization or chemical inhibition of the cell cycle. Methods and Materials: Two related cell lines with disparate HRS status were used (MR4 and 3.7 cells). A double-thymidine block technique was developed to enrich the G2-phase population. Clonogenic cell survival, radiation-induced G2-phase cell cycle arrest, and deoxyribonucleic acid double-strand break repair were measured in the presence and absence of inhibitors to G2-phase checkpoint proteins. Results: For MR4 cells, the dose required to overcome the HRS response (approximately 0.2 Gy) corresponded with that needed for the activation of the early G2/M checkpoint. As hypothesized, enriching the number of G2-phase cells in the population resulted in an enhanced HRS response, because a greater proportion of radiation-damaged cells evaded the early G2/M checkpoint and entered mitosis with unrepaired deoxyribonucleic acid double-strand breaks. Likewise, abrogation of the checkpoint by inhibition of Chk1 and Chk2 also increased low-dose radiosensitivity. These effects were not evident in 3.7 cells. Conclusions: The data confirm that HRS is linked to the early G2/M checkpoint through the damage response of G2-phase cells. Low-dose radiosensitivity could be increased by manipulating the transition of radiation-damaged G2-phase cells into mitosis. This provides a rationale for combining low-dose radiation therapy with chemical synchronization techniques to improve increased radiosensitivity.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Compiler-Enhanced Incremental Checkpointing for OpenMP Applications

As modern supercomputing systems reach the peta-flop performance range, they grow in both size and complexity. This makes them increasingly vulnerable to failures from a variety of causes. Checkpointing is a popular technique for tolerating such failures, enabling applications to periodically save their state and restart computation after a failure. Although a variety of automated system-level checkpointing solutions are currently available to HPC users, manual application-level checkpointing remains more popular due to its superior performance. This paper improves performance of automated checkpointing via a compiler analysis for incremental checkpointing. This analysis, which works with both sequential and OpenMP applications, significantly reduces checkpoint sizes and enables asynchronous checkpointing.

99 GENERAL AND MISCELLANEOUS↗

Understanding checkpointing overheads on massive-scale systems : analysis of the IBM Blue Gene/P system.

Providing fault tolerance in high-end petascale systems, consisting of millions of hardware components and complex software stacks, is becoming an increasingly challenging task. Checkpointing continues to be the most prevalent technique for providing fault tolerance in such high-end systems. Considerable research has focussed on optimizing checkpointing; however, in practice, checkpointing still involves a high-cost overhead for users. In this paper, we study the checkpointing overhead seen by various applications running on leadership-class machines like the IBM Blue Gene/P at Argonne National Laboratory. In addition to studying popular applications, we design a methodology to help users understand and intelligently choose an optimal checkpointing frequency to reduce the overall checkpointing overhead incurred. In particular, we study the Grid-Based Projector-Augmented Wave application, the Carr-Parrinello Molecular Dynamics application, the Nek5000 computational fluid dynamics application and the Parallel Ocean Program application-and analyze their memory usage and possible checkpointing trends on 65,536 processors of the Blue Gene/P system.

99 GENERAL AND MISCELLANEOUS↗

Using the Sirocco File System for high-bandwidth checkpoints.

The Sirocco File System, a file system for exascale under active development, is designed to allow the storage software to maximize quality of service through increased flexibility and local decision-making. By allowing the storage system to manage a range of storage targets that have varying speeds and capacities, the system can increase the speed and surety of storage to the application. We instrument CTH to use a group of RAM-based Sirocco storage servers allocated within the job as a high-performance storage tier to accept checkpoints, allowing computation to potentially continue asynchronously of checkpoint migration to slower, more permanent storage. The result is a 10-60x speedup in constructing and moving checkpoint data from the compute nodes. This demonstration of early Sirocco functionality shows a significant benefit for a real I/O workload, checkpointing, in a real application, CTH. By running Sirocco storage servers within a job as RAM-only stores, CTH was able to store checkpoints 10-60x faster than storing to PanFS, allowing the job to continue computing sooner. While this prototype did not include automatic data migration, the checkpoint was available to be pushed or pulled to disk-based storage as needed after the compute nodes continued computing. Future developments include the ability to dynamically spawn Sirocco nodes to absorb checkpoints, expanding this mechanism to other fast tiers of storage like flash memory, and sharing of dynamic Sirocco nodes between multiple jobs as needed.

99 GENERAL AND MISCELLANEOUS↗

REMEM: REmote MEMory as Checkpointing Storage

Checkpointing is a widely used mechanism for supporting fault tolerance, but notorious in its high-cost disk access. The idea of memory-based checkpointing has been extensively studied in research but made little success in practice due to its complexity and potential reliability concerns. In this study we present the design and implementation of REMEM, a REmote MEMory checkpointing system to extend the checkpointing storage from disk to remote memory. A unique feature of REMEM is that it can be integrated into existing disk-based checkpointing systems seamlessly. A user can flexibly switch between REMEM and disk as checkpointing storage to balance the efficiency and reliability. The implementation of REMEM on Open MPI is also introduced. The experimental results confirm that REMEM and the proposed adaptive checkpointing storage selection are promising in both performance, reliability and scalability.

97 MATHEMATICS AND COMPUTING↗

Checkpoint triggering in a computer system

According to an aspect, a method for triggering creation of a checkpoint in a computer system includes executing a task in a processing node and determining whether it is time to read a monitor associated with a metric of the task. The monitor is read to determine a value of the metric based on determining that it is time to read the monitor. A threshold for triggering creation of the checkpoint is determined based on the metric. A monitoring block size is determined for the checkpoint. A checkpoint interval is determined based on the monitoring block size, a checkpoint bandwidth, and a failure rate of the computer system. Based on determining that the value of the metric has crossed the threshold and the checkpoint interval has elapsed, the checkpoint including state data of the task is created to enable restarting execution of the task upon a restart operation.

97 MATHEMATICS AND COMPUTING↗

Checkpoint triggering in a computer system

According to an aspect, a method for triggering creation of a checkpoint in a computer system includes executing a task in a processing node of the computer system. A monitoring block size is determined for the checkpoint. A checkpoint interval is determined based on the monitoring block size, a checkpoint bandwidth, and a failure rate of the computer system. Based on determining that the checkpoint interval has elapsed, the checkpoint including state data of the task is created to enable restarting execution of the task upon a restart operation. The state data of the checkpoint is restored from a memory responsive to detecting an error condition at the processing node. Execution of the task is restarted in the processing node based on the state data restored from the memory.

97 MATHEMATICS AND COMPUTING↗

Lyn tyrosine kinase promotes silencing of ATM-dependent checkpoint signaling during recovery from DNA double-strand breaks

Highlights: • Inhibition of Src family kinases decreased γ-H2AX signal. • Inhibition of Src family increased ATM-dependent phosphorylation of Chk2 and Kap1. • shRNA-mediated knockdown of Lyn increased phosphorylation of Kap1 by ATM. • Ectopic expression of Src family kinase suppressed ATM-mediated Kap1 phosphorylation. • Src is involved in upstream signaling for inactivation of ATM signaling. - Abstract: DNA damage activates the DNA damage checkpoint and the DNA repair machinery. After initial activation of DNA damage responses, cells recover to their original states through completion of DNA repair and termination of checkpoint signaling. Currently, little is known about the process by which cells recover from the DNA damage checkpoint, a process called checkpoint recovery. Here, we show that Src family kinases promote inactivation of ataxia telangiectasia mutated (ATM)-dependent checkpoint signaling during recovery from DNA double-strand breaks. Inhibition of Src activity increased ATM-dependent phosphorylation of Chk2 and Kap1. Src inhibition increased ATM signaling both in G2 phase and during asynchronous growth. shRNA knockdown of Lyn increased ATM signaling. Src-dependent nuclear tyrosine phosphorylation suppressed ATM-mediated Kap1 phosphorylation. These results suggest that Src family kinases are involved in upstream signaling that leads to inactivation of the ATM-dependent DNA damage checkpoint.

60 APPLIED LIFE SCIENCES↗

Structural basis for the activity and specificity of the immune checkpoint inhibitor lirilumab

The clinical success of immune checkpoint inhibitors has underscored the key role of the immune system in controlling cancer. Current FDA-approved immune checkpoint inhibitors target the regulatory receptor pathways of cytotoxic T-cells to enhance their anticancer responses. Despite an abundance of evidence that natural killer (NK) cells can also mediate potent anticancer activities, there are no FDA-approved inhibitors targeting NK cell specific checkpoint pathways. Lirilumab, the most clinically advanced NK cell checkpoint inhibitor, targets inhibitory killer immunoglobulin-like receptors (KIRs), however it has yet to conclusively demonstrate clinical efficacy. Here we describe the crystal structure of lirilumab in complex with the inhibitory KIR2DL3, revealing the precise epitope of lirilumab and the molecular mechanisms underlying KIR checkpoint blockade. Notably, the epitope includes several key amino acids that vary across the human population, and binding studies demonstrate the importance of these amino acids for lirilumab binding. These studies reveal how KIR variations in patients could influence the clinical efficacy of lirilumab and reveal general concepts for the development of immune checkpoint inhibitors targeting NK cells.

60 APPLIED LIFE SCIENCES↗

Reducing space overhead for independendent checkpointing

The main disadvantages of independent checkpointing are the possible domino effect and the associated storage space overhead for maintaining multiple checkpoints. In most previous work, it has been assumed that only the checkpoints older than the current global recovery line can be discarded. Here, we generalize a notion of recovery line to potential recovery line. Only the checkpoints belonging to at least one of the potential recovery lines cannot be discarded. By using the model of maximum-sized antichains on a partially ordered set, an efficient algorithm is developed for finding all non-discardable checkpoints, and we show that the number of non-discardable checkpoints cannot exceed N(N+1)/2, where N is the number of processors. Communication trace driven simulation for several hypercube programs is performed to show the benefit of the proposed algorithm for real applications.

Wang, Yi-Min↗

Lazy checkpoint coordination for bounding rollback propagation

Independent checkpointing allows maximum process autonomy but suffers from potential domino effects. Coordinated checkpointing eliminates the domino effect by sacrificing a certain degree of process autonomy. In this paper, we propose the technique of lazy checkpoint coordination which preserves process autonomy while employing communication-induced checkpoint coordination for bounding rollback propagation. The introduction of the notion of laziness allows a flexible trade-off between the cost for checkpoint coordination and the average rollback distance. Worst-case overhead analysis provides a means for estimating the extra checkpoint overhead. Communication trace-driven simulation for several parallel programs is used to evaluate the benefits of the proposed scheme for real applications.

Wang, Yi-Min↗

Compiler-assisted static checkpoint insertion

This paper describes a compiler-assisted approach for static checkpoint insertion. Instead of fixing the checkpoint location before program execution, a compiler enhanced polling mechanism is utilized to maintain both the desired checkpoint intervals and reproducible checkpoint 1ocations. The technique has been implemented in a GNU CC compiler for Sun 3 and Sun 4 (Sparc) processors. Experiments demonstrate that the approach provides for stable checkpoint intervals and reproducible checkpoint placements with performance overhead comparable to a previously presented compiler assisted dynamic scheme (CATCH) utilizing the system clock.

Long, Junsheng↗

Lazy checkpoint coordination for bounding rollback propagation

Independent checkpointing allows maximum process autonomy but suffers from potential domino effects. Coordinated checkpointing eliminates the domino effect by sacrificing a certain degree of process autonomy. In this paper, we propose the technique of lazy checkpoint coordination which preserves process autonomy while employing communication-induced checkpoint coordination for bounding rollback propagation. The introduction of the notion of laziness allows a flexible trade-off between the cost for checkpoint coordination and the average rollback distance. Worst-case overhead analysis provides a means for estimating the extra checkpoint overhead. Communication trace-driven simulation for several parallel programs is used to evaluate the benefits of the proposed scheme for real applications.

Wang, Yi-Min↗

Template based parallel checkpointing in a massively parallel computer system

A method and apparatus for a template based parallel checkpoint save for a massively parallel super computer system using a parallel variation of the rsync protocol, and network broadcast. In preferred embodiments, the checkpoint data for each node is compared to a template checkpoint file that resides in the storage and that was previously produced. Embodiments herein greatly decrease the amount of data that must be transmitted and stored for faster checkpointing and increased efficiency of the computer system. Embodiments are directed to a parallel computer system with nodes arranged in a cluster with a high speed interconnect that can perform broadcast communication. The checkpoint contains a set of actual small data blocks with their corresponding checksums from all nodes in the system. The data blocks may be compressed using conventional non-lossy data compression algorithms to further reduce the overall checkpoint size.

97 MATHEMATICS AND COMPUTING↗

DeepFreeze: Towards Scalable Asynchronous Checkpointing of Deep Learning Models

In the age of big data, deep learning has emerged as a powerful tool to extract insight and exploit its value, both in industry and scientific applications. One common pattern emerging in such applications is frequent checkpointing of the state of the learning model during training, needed in a variety of scenarios: analysis of intermediate states to explain features and correlations with training data, exploration strategies involving alternative models that share a common ancestor, knowledge transfer, resilience, etc. However, with increasing size of the learning models and popularity of distributed data-parallel training approaches, simple checkpointing techniques used so far face several limitations: low serialization performance, blocking I/O, stragglers due to the fact that only a single process is involved in checkpointing. This paper proposes a checkpointing technique specifically designed to address the aforementioned limitations, introducing efficient asynchronous techniques to hide the overhead of serialization and I/O, and distribute the load over all participating processes. Experiments with two deep learning applications (CANDLE and ResNet) on a pre-Exascale HPC platform (Theta) shows significant improvement over state-of-art, both in terms of checkpointing duration and runtime overhead.

checkpointing↗