Search NASA⌕ Search

SEARCH · Search NASA

Results for “checkpointing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

Scalable Transparent Checkpoint-Restart of Global Address Space Applications on Virtual Machines over Infiniband

Checkpoint-Restart is one of the most used software approaches to achieve fault-tolerance in high-end clusters. While standard techniques typically focus on user-level solutions, the advent of virtualization software has enabled efficient and transparent system-level approaches. In this paper, we present a scalable transparent system-level solution to address fault-tolerance for applications based on global address space (GAS) programming models on Infiniband clusters. In addition to handling communication, the solution addresses transparent checkpoint of user-generated files. We exploit the support for the Infiniband network in the Xen virtual machine environment. We have developed a version of the Aggregate Remote Memory Copy Interface (ARMCI) one-sided communication library capable of suspending and resuming applications. We present efficient and scalable mechanisms to distribute checkpoint requests and to backup virtual machines memory images and file systems. We tested our approach in the context of NWChem, a popular computational chemistry suite. We demonstrated that NWChem can be executed, without any modification to the source code, on a virtualized 8-node cluster with very little overhead (below 3%). We observe that the total checkpoint time is limited by disk I/O. Finally, we measured system-size depended components of the checkpoint time on up to 1024 cores (128 nodes), demonstrating the scalability of our approach in medium/large-scale systems.

97 MATHEMATICS AND COMPUTING↗

Leveraging the checkpoint-restart technique for optimizing CPU efficiency of ATLAS production applications on opportunistic platforms

Data processing applications of the ATLAS experiment, such as event simulation and reconstruction, spend considerable amount of time in the initialization phase. This phase includes loading a large number of shared libraries, reading detector geometry and condition data from external databases, building a transient representation of the detector geometry and initializing various algorithms and services. In some cases the initialization step can take as long as 10-15 minutes. Such slow initialization has a significant negative impact on overall CPU efficiency of the production job, especially when the job is executed on opportunistic, often short-lived, resources such as commercial clouds or volunteer computing. In order to improve this situation, we can take advantage of the fact that ATLAS runs large numbers of production jobs with similar configuration parameters (e.g. jobs within the same production task). This allows us to checkpoint one job at the end of its configuration step and then use the generated checkpoint image for rapid startup of thousands of production jobs. By applying this technique we can bring the initialization time of a job from tens of minutes down to just a few seconds. In addition to that we can leverage container technology for restarting checkpointed applications on the variety of computing platforms, in particular of platforms different from the one on which the checkpoint image was created. We will describe the mechanism of creating checkpoint images of Geant4 simulation jobs with AthenaMP (the multi-process version of the ATLAS data simulation, reconstruction and analysis framework Athena) and the usage of these images for running ATLAS Simulation production jobs on volunteer computing resources (ATLAS@Home) and on Supercomputers.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Benchmarking Variables for Checkpointing in HPC Applications

Checkpoint/Restart (C/R) is a widely used fault tolerance mechanism in converged systems of cloud, edge, and HPC. However, users often rely on their experience to determine which variables to checkpoint, as there is currently no benchmark that can provide a reference. This can result in checkpointing redundant or even incorrect variables. To address this issue, we propose a benchmark suite that includes critical variables for checkpointing, which have been manually identified, and a method for identifying those critical variables, with 20 representative HPC applications. Our method involves analyzing data dependency between variables to identify critical variables analytically. We verify the identified variables' correctness with a widely used C/R library FTI by an ablation study. With our benchmark suite and data dependency analysis, HPC practitioners now have a reference for identifying checkpointing variables and better knowledge of what kind of variables to checkpoint.

Fu, Xiang↗

An Efficient Checkpointing System for Large Machine Learning Model Training

As machine learning models increase in size and complexity rapidly, the cost of checkpointing in ML training became a bottleneck in storage and performance (time). For example, the latest GPT-4 model has massive parameters at the scale of 1.76 trillion. It is highly time and storage consuming to frequently writes the model to checkpoints with more than 1 trillion floating point values to storage. This work aims to understand and attempt to mitigate this problem. First, we characterize the checkpointing interface in a collection of representative large machine learning/language models with respect to storage consumption and performance overhead. Second, we propose the two optimizations: i) A periodic cleaning strategy that periodically cleans up outdated checkpoints to reduce the storage burden; ii) A data staging optimization that coordinates checkpoints between local and shared file systems for performance improvement.

machine learning, artificial intelligence↗

Non-volatile memory for checkpoint storage

A system, method and computer program product for supporting system initiated checkpoints in high performance parallel computing systems and storing of checkpoint data to a non-volatile memory storage device. The system and method generates selective control signals to perform checkpointing of system related data in presence of messaging activity associated with a user application running at the node. The checkpointing is initiated by the system such that checkpoint data of a plurality of network nodes may be obtained even in the presence of user applications running on highly parallel computers that include ongoing user messaging activity. In one embodiment, the non-volatile memory is a pluggable flash memory card.

97 MATHEMATICS AND COMPUTING↗

Identifying security checkpoints locations to protect the major U.S. urban areas

Transit networks are integral to the economy and to society, but at the same time they could allow terrorists to transport weapons of mass destruction into any city. Road networks are especially vulnerable, because they lack natural checkpoints unlike air networks that have security measures in place at all major airports. One approach to mitigate this risk is ensuring that every road route passes through at least one security checkpoint. Using the Ford-Fulkerson maximum-flow algorithm, we generate a minimum set of checkpoint locations within a ring-shaped buffer area surrounding the 50 largest US urban areas. We study how the number of checkpoints changes as we increase the buffer width to perform a cost-benefit analysis and to identify groups of cities that behave similarly. The set of required checkpoints is surprisingly small (10-124) despite the hundreds of thousands of road arcs in those areas, making it feasible to protect all major cities.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

Checkpointing for a hybrid computing node

According to an aspect, a method for checkpointing in a hybrid computing node includes executing a task in a processing accelerator of the hybrid computing node. A checkpoint is created in a local memory of the processing accelerator. The checkpoint includes state data to restart execution of the task in the processing accelerator upon a restart operation. Execution of the task is resumed in the processing accelerator after creating the checkpoint. The state data of the checkpoint are transferred from the processing accelerator to a main processor of the hybrid computing node while the processing accelerator is executing the task.

97 MATHEMATICS AND COMPUTING↗

Analyzing checkpointing trends for applications on the IBM Blue Gene/P system.

Current petascale systems have tens of thousands of hardware components and complex system software stacks, which increase the probability of faults occurring during the lifetime of a process. Checkpointing has been a popular method of providing fault tolerance in high-end systems. While considerable research has been done to optimize checkpointing, in practice the method still involves a high-cost overhead for users. In this paper, we study the checkpointing overhead seen by applications running on leadership-class machines such as the IBM Blue Gene/P at Argonne National Laboratory. We study various applications and design a methodology to assist users in understanding and choosing checkpointing frequency and reducing the overhead incurred. In particular, we study three popular applications -- the Grid-Based Projector-Augmented Wave application, the Carr-Parrinello Molecular Dynamics application, and a Nek5000 computational fluid dynamics application -- and analyze their memory usage and possible checkpointing trends on 32,768 processors of the Blue Gene/P system.

97 MATHEMATICS AND COMPUTING↗

Coping with silent and fail-stop errors at scale by combining replication and checkpointing

This paper provides a model and an analytical study of replication as a technique to cope with silent errors, as well as a mixture of both silent and fail-stop errors on large-scale platforms. Compared with fail-stop errors that are immediately detected when they occur, silent errors require a detection mechanism. To detect silent errors, many application-specific techniques are available, either based on algorithms (e.g., ABFT), invariant preservation or data analytics, but replication remains the most transparent and least intrusive technique. We explore the right level (duplication, triplication or more) of replication for two frameworks: (i) when the platform is subject to only silent errors, and (ii) when the platform is subject to both silent and fail-stop errors. A higher level of replication is more expensive in terms of resource usage but enables to tolerate more errors and to even correct some errors, hence there is a trade-off to be found. Replication is combined with checkpointing and comes with two flavors: process replication and group replication. Process replication applies to message-passing applications with communicating processes. Each process is replicated, and the platform is composed of process pairs, or triplets. Group replication applies to black-box applications, whose parallel execution is replicated several times. The platform is partitioned into two halves (or three thirds). In both scenarios, results are compared before each checkpoint, which is taken only when both results (duplication) or two out of three results (triplication) coincide. Otherwise, one or more silent errors have been detected, and the application rolls back to the last checkpoint, as well as when fail-stop errors have struck. We provide a detailed analytical study for all of these scenarios, with formulas to decide, for each scenario, the optimal parameters as a function of the error rate, checkpoint cost, and platform size. We also report a set of extensive simulation results that nicely corroborates the analytical model.

Checkpointing↗

A cancer-associated polymorphism in ESCRT-III disrupts the abscission checkpoint and promotes genome instability

Cytokinetic abscission facilitates the irreversible separation of daughter cells. This process requires the endosomal-sorting complexes required for transport (ESCRT) machinery and is tightly regulated by charged multivesicular body protein 4C (CHMP4C), an ESCRT-III subunit that engages the abscission checkpoint (NoCut) in response to mitotic problems such as persisting chromatin bridges within the midbody. Importantly, a human polymorphism in CHMP4C (rs35094336, CHMP4C T232 ) increases cancer susceptibility. Here, we explain the structural and functional basis for this cancer association: The CHMP4C T232 allele unwinds the C-terminal helix of CHMP4C, impairs binding to the early-acting ESCRT factor ALIX, and disrupts the abscission checkpoint. Cells expressing CHMP4C T232 exhibit increased levels of DNA damage and are sensitized to several conditions that increase chromosome missegregation, including DNA replication stress, inhibition of the mitotic checkpoint, and loss of p53. Our data demonstrate the biological importance of the abscission checkpoint and suggest that dysregulation of abscission by CHMP4C T232 may synergize with oncogene-induced mitotic stress to promote genomic instability and tumorigenesis.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Casein kinase II is required for the spindle assembly checkpoint by regulating Mad2p in fission yeast

The spindle checkpoint is a surveillance mechanism that ensures the fidelity of chromosome segregation in mitosis. Here we show that fission yeast casein kinase II (CK2) is required for this checkpoint function. In the CK2 mutants mitosis occurs in the presence of a spindle defect, and the spindle checkpoint protein Mad2p fails to localize to unattached kinetochores. The CK2 mutants are sensitive to the microtubule depolymerising drug thiabendazole, which is counteracted by ectopic expression of mad2{sup +}. The level of Mad2p is low in the CK2 mutants. These results suggest that CK2 has a role in the spindle checkpoint by regulating Mad2p.

60 APPLIED LIFE SCIENCES↗

Physics-aware adaptive checkpointing with shadow systems for nonlinear PDE simulations

Large-scale simulations of nonlinear partial differential equations (PDEs) that exhibit strongly transient behavior and pattern-forming dynamics produce enormous amounts of data, which, even with modern storage systems, cannot be stored for later curation. Current I/O strategies either write dense time series of snapshots, which is often prohibitive in I/O and storage, or store a few checkpoints that enable restart but incur expensive recomputation cost and provide no control over post-restart error growth, especially when lossy compression is used. Moreover, most, if not all, existing strategies take no account of the actual physical state of the system. Here, we present a simple physics-aware I/O framework in which a low-cost shadow system adaptively triggers lossy checkpoints when the shadow system deviates from the fine-scale simulation. The shadow system can be a coarsened replica of the fine-scale simulation that evolves concurrently. This means that checkpoints are taken based on the physical state of the system: fewer checkpoints are triggered when the system is quiescent while more are taken when the system undergoes a rapid change. This type of behavior is observed in many systems such as Brusselator and FitzHugh–Nagumo. We illustrate that our framework maintains stable restarts, keeps fine-scale restart errors bounded by shadow errors, and reconstructs the time history with significantly lower error and storage than interpolating fixed-interval snapshots, with low-cost shadow replay and modest online synchronization overhead.

Gong, Qian [ORNL] (ORCID:0000000235704142)↗

Structure and Substrate Recruitment of the Human Spindle Checkpoint Kinase Bub1

In mitosis, the spindle checkpoint detects a single unattached kinetochore, inhibits the anaphase-promoting complex or cyclosome (APC/C), and prevents premature sister chromatid separation. The checkpoint kinase Bub1 contributes to checkpoint sensitivity through phosphorylating the APC/C activator, Cdc20, and inhibiting APC/C catalytically. We report here the crystal structure of the kinase domain of Bub1, revealing the requirement of an N-terminal extension for its kinase activity. Though the activation segment of Bub1 is ordered and has structural features indicative of active kinases, the C-terminal portion of this segment sterically restricts substrate access to the active site. Bub1 uses docking motifs, so-called KEN boxes, outside its kinase domain to recruit Cdc20, one of two known KEN box receptors. The KEN boxes of Bub1 are required for the spindle checkpoint in human cells. Therefore, its unusual active-site conformation and mode of substrate recruitment suggest that Bub1 has an exquisitely tuned specificity for Cdc20.

36 MATERIALS SCIENCE↗

Mechanistic insight into TRIP13-catalyzed Mad2 structural transition and spindle checkpoint silencing

The spindle checkpoint maintains genomic stability and prevents aneuploidy. Unattached kinetochores convert the latent open conformer of the checkpoint protein Mad2 (O-Mad2) to the active closed conformer (C-Mad2), bound to Cdc20. C-Mad2–Cdc20 is incorporated into the mitotic checkpoint complex (MCC), which inhibits the anaphase-promoting complex/cyclosome (APC/C). The C-Mad2-binding protein p31 comet and the ATPase TRIP13 promote MCC disassembly and checkpoint silencing. Here, using nuclear magnetic resonance (NMR) spectroscopy, we show that TRIP13 and p31 comet catalyze the conversion of C-Mad2 to O-Mad2, without disrupting its stably folded core. We determine the crystal structure of human TRIP13, and identify functional TRIP13 residues that mediate p31 comet –Mad2 binding and couple ATP hydrolysis to local unfolding of Mad2. TRIP13 and p31 comet prevent APC/C inhibition by MCC components, but cannot reactivate APC/C already bound to MCC. Therefore, TRIP13–p31 comet intercepts and disassembles free MCC not bound to APC/C through mediating the local unfolding of the Mad2 C-terminal region.

59 BASIC BIOLOGICAL SCIENCES↗

Quantification, Trade-off Analysis, and Optimal Checkpoint Placement for Reliability and Availability

Checkpointing is the most widely used technique in high-performance computing (HPC) to ensure the application progress in the presence of failures. In this paper, we present mathematical models of the checkpointing systems to quantify their reliability and availability. We perform trade-off analysis with respect to resource costs and reliability. Then, we explore the optimal checkpoint placement for checkpointing systems to maximize system availability. Finally, in a rigorous manner, we comparatively analyze the behavior of redundant systems where replication and repair mechanisms are employed. We postulate that the proposed models can aid system designers, who can instantiate our models to assess and quantify the availability and reliability of systems of interest. Our study demonstrates that the configuration with most reliable and available systems depends on the parameter settings.

Subasi, Omer↗

AutoCheck: Automatically Identifying Variables for Checkpointing by Data Dependency Analysis

Checkpoint/Restart (C/R) has been widely deployed in numerous HPC systems, Clouds, and industrial data centers, which are typically operated by system engineers. Nevertheless, there is no existing approach that helps system engineers without domain expertise and domain scientists without system fault tolerance knowledge identify those critical variables accounted for correct application execution restoration in a failure for C/R. To address this problem, we propose an analytical model and a tool (AutoCheck) that can automatically identify critical variables to checkpoint for C/R. AutoCheck relies on first, analytically tracking and optimizing data dependency between variables and other application execution state, and second, a set of heuristics that identify critical variables for checkpointing from the refined data dependency graph (DDG). AutoCheck allows programmers to pinpoint critical variables to checkpoint quickly within a few minutes. We evaluate AutoCheck on 13 representative HPC benchmarks, demonstrating that AutoCheck can efficiently identify correct critical variables to checkpoint.

HPC↗

Scrutinizing Variables for Checkpoint Using Automatic Differentiation

Checkpoint/Restart (C/R) saves the running state of the programs periodically, which consumes considerable time and system resources. We observe that not every piece of data is involved in the computation in typical HPC applications; such unused data should be excluded from checkpointing for better storage and compute efficiency. We propose a systematic approach that leverages automatic differentiation (AD) to scrutinize every element within variables (e.g., arrays) necessary for checkpointing. This allows us to identify critical and uncritical elements and eliminate uncritical elements from checkpointing. Specifically, we inspect every single element within a variable necessary for checkpointing with an AD tool to determine whether the element has an impact on the application output or not. We validate our approach with all benchmarks from the NPB suite. We visualize the distribution of critical and uncritical elements within a variable with respect to its binary impact (yes or no) on the application output.

Huang, Xin [Kobe University]↗

MPI Stages: Checkpointing MPI State for Bulk Synchronous Applications

When an MPI program experiences a failure, the most common recovery approach is to restart all processes from a previous checkpoint and to re-queue the entire job. A disadvantage of this method is that, although the failure occurred within the main application loop, live processes must start again from the beginning of the program, along with new replacement processes---this incurs unnecessary overhead for live processes. To avoid such overheads and concomitant delays, we introduce the concept of "MPI Stages." MPI Stages saves internal MPI state in a separate checkpoint in conjunction with application state. Upon failure, both MPI and application state are recovered, respectively, from their last synchronous checkpoints and continue without restarting the overall MPI job. Live processes roll back only a few iterations within the main loop instead of rolling back to the beginning of the program, while a replacement of failed process restarts and reintegrates, thereby achieving faster failure recovery. This approach integrates well with large-scale, bulk synchronous applications and checkpoint/restart. In this paper, we identify requirements for production MPI implementations to support state checkpointing with MPI Stages, which includes capturing and managing internal MPI state and serializing and deserializing user handles to MPI objects. We evaluate our fault tolerance approach with a proof-of-concept prototype MPI implementation that includes MPI Stages. We demonstrate its functionality and performance using LULESH and microbenchmarks. Our results show that MPI Stages reduces the recovery time by 13× for LULESH in comparison to checkpoint/restart.

Sultana, Nawrin↗