Search NASA⌕ Search

SEARCH · Search NASA

Results for “checkpointing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Optimal message log reclamation for uncoordinated checkpointing

Uncoordinated checkpointing for message-passing systems allows maximum process autonomy and general nondeterministic execution, but suffers from potential domino effect and the large space overhead for maintaining checkpoints and message logs. Traditionally, it has been assumed that only obsolete checkpoints and message logs before the global recovery line can be garbage-collected. Recently, an approach to identifying all garbage checkpoints based on recovery line transformation and decomposition has been developed. We show in this paper that the same approach can be applied to the problem of identifying all garbage message logs for systems requiring message logging to record in-transit messages. Communication trace-driven simulation for several parallel programs is used to evaluate the proposed algorithm.

Wang, Yi-Min↗

An ATM-independent S-phase checkpoint response involves CHK1 pathway

After exposure to genotoxic stress, proliferating cells actively slow down the DNA replication through a S-phase checkpoint to provide time for repair. We report that in addition to the ataxia-telangiectasia mutated (ATM)-dependent pathway that controls the fast response, there is an ATM-independent pathway that controls the slow response to regulate the S-phase checkpoint after ionizing radiation in mammalian cells. The slow response of S-phase checkpoint, which is resistant to wortmannin, sensitive to caffeine and UCN-01, and related to cyclin-dependent kinase phosphorylation, is much stronger in CHK1 overexpressed cells, and it could be abolished by Chk1 antisense oligonucleotides. These results provide evidence that the ATM-independent slow response of S-phase checkpoint involves CHK1 pathway.

Non-NASA Center↗

Network support for system initiated checkpoints

A system, method and computer program product for supporting system initiated checkpoints in parallel computing systems. The system and method generates selective control signals to perform checkpointing of system related data in presence of messaging activity associated with a user application running at the node. The checkpointing is initiated by the system such that checkpoint data of a plurality of network nodes may be obtained even in the presence of user applications running on highly parallel computers that include ongoing user messaging activity.

97 MATHEMATICS AND COMPUTING↗

Checkpointing in speculative versioning caches

Mechanisms for generating checkpoints in a speculative versioning cache of a data processing system are provided. The mechanisms execute code within the data processing system, wherein the code accesses cache lines in the speculative versioning cache. The mechanisms further determine whether a first condition occurs indicating a need to generate a checkpoint in the speculative versioning cache. The checkpoint is a speculative cache line which is made non-speculative in response to a second condition occurring that requires a roll-back of changes to a cache line corresponding to the speculative cache line. The mechanisms also generate the checkpoint in the speculative versioning cache in response to a determination that the first condition has occurred.

97 MATHEMATICS AND COMPUTING↗

Network support for system initiated checkpoints

A system, method and computer program product for supporting system initiated checkpoints in parallel computing systems. The system and method generates selective control signals to perform checkpointing of system related data in presence of messaging activity associated with a user application running at the node. The checkpointing is initiated by the system such that checkpoint data of a plurality of network nodes may be obtained even in the presence of user applications running on highly parallel computers that include ongoing user messaging activity.

Chen, Dong↗

Orchestrating Fault Prediction with Live Migration and Checkpointing

Checkpoint/Restart (C/R) is widely used to provide fault tolerance on High-Performance Computing (HPC) systems. However, Parallel File System (PFS) overhead and failure uncertainty cause significant application overhead. This paper develops an adaptive multi-level C/R model that incorporates a failure prediction and analysis model, which orchestrates failure prediction, checkpointing, checkpoint frequency, and proactive live migration along with the additional benefit of Burst Buffers (BB). It effectively reduces the overheads due to failures, checkpointing, and recovery. Simulation results for the Summit supercomputer yield a reduction of ~20%-86% in application overhead due to BBs, orchestrated failure prediction, and migration. We also observe a ~29% decrease in checkpoint writes to BBs, which can increase the longevity of the BB storage devices.

Behera, Subhendu↗

Adaptive multi-level checkpointing

In some examples, with respect to adaptive multi-level checkpointing, a transfer parameter associated with transfer of checkpoint data from a node-local storage to a parallel file system may be ascertained for the checkpoint data stored in the node-local storage. The transfer parameter may be compared to a specified transfer parameter threshold. A determination may be made, based on the comparison of the transfer parameter to the specified transfer parameter threshold, as to whether to transfer the checkpoint data from the node-local storage to the parallel file system.

97 MATHEMATICS AND COMPUTING↗

Node failure resiliency for Uintah without checkpointing

The frequency of failures in upcoming exascale supercomputers may well be greater than at present due to many-core architectures if component failure rates remain unchanged. This potential increase in failure frequency coupled with I/O challenges at exascale may prove problematic for current resiliency approaches such as checkpoint restarting, although the use of fast intermediate memory may help. Algorithm-Based Fault Tolerance (ABFT) using Adaptive Mesh Refinement (AMR) is one resiliency approach used to address these challenges. For adaptive mesh codes, a coarse mesh version of the solu- tion may be used to restore the fine mesh solution. This paper addresses the implementation of the ABFT approach within the Uintah software framework: both at a software level within Uintah and in the data reconstruction method used for the recovery of lost data. This method has two problems: inaccuracies introduced during the reconstruction propagate forward in time, and the physical consistency of variables such as positivity or boundedness may be violated during interpolation. These challenges can be addressed by the combination of two techniques: 1. a fault-tolerant MPI implementation to recover from runtime node failures, and 2. high-order interpolation schemes to preserve the physical solution and reconstruct lost data. Here, the approach considered here uses a "Limited Essentially Non-Oscillatory" (LENO) scheme along with AMR to rebuild the lost data without checkpointing using Uintah. Experiments were carried out using a fault-tolerant MPI - ULFM to recover from runtime failure, and LENO to recover data on patches belonging to failed ranks, while the simulation was continued to the end. Results show that this ABFT approach is up to 10x faster than the traditional checkpointing method. The new interpolation approach is more accurate than linear interpolation and not subject to the overshoots found in other interpolation methods.

97 MATHEMATICS AND COMPUTING↗

A checkpoint compression study for high-performance computing systems

As high-performance computing systems continue to increase in size and complexity, higher failure rates and increased overheads for checkpoint/restart (CR) protocols have raised concerns about the practical viability of CR protocols for future systems. Previously, compression has proven to be a viable approach for reducing checkpoint data volumes and, thereby, reducing CR protocol overhead leading to improved application performance. In this article, we further explore compression-based CR optimization by exploring its baseline performance and scaling properties, evaluating whether improved compression algorithms might lead to even better application performance and comparing checkpoint compression against and alongside other software- and hardware-based optimizations. Our results highlights are: (1) compression is a very viable CR optimization; (2) generic, text-based compression algorithms appear to perform near optimally for checkpoint data compression and faster compression algorithms will not lead to better application performance; (3) compression-based optimizations fare well against and alongside other software-based optimizations; and (4) while hardware-based optimizations outperform software-based ones, they are not as cost effective.

97 MATHEMATICS AND COMPUTING↗

Lossy checkpoint compression in full waveform inversion: a case study with ZFPv0.5.5 and the overthrust model

This paper proposes a new method that combines checkpointing methods with error-controlled lossy compression for large-scale high-performance full-waveform inversion (FWI), an inverse problem commonly used in geophysical exploration. This combination can significantly reduce data movement, allowing a reduction in run time as well as peak memory. In the exascale computing era, frequent data transfer (e.g., memory bandwidth, PCIe bandwidth for GPUs, or network) is the performance bottleneck rather than the peak FLOPS of the processing unit. Like many other adjoint-based optimization problems, FWI is costly in terms of the number of floating-point operations, large memory footprint during backpropagation, and data transfer overheads. Past work for adjoint methods has developed checkpointing methods that reduce the peak memory requirements during backpropagation at the cost of additional floating-point computations. Combining this traditional checkpointing with error-controlled lossy compression, we explore the three-way tradeoff between memory, precision, and time to solution. We investigate how approximation errors introduced by lossy compression of the forward solution impact the objective function gradient and final inverted solution. Empirical results from these numerical experiments indicate that high lossy-compression rates (compression factors ranging up to 100) have a relatively minor impact on convergence rates and the quality of the final solution.

58 GEOSCIENCES↗

Characterization of a cancer cell line that expresses a splicing variant form of 53BP1: Separation of checkpoint and repair functions in 53BP1

53BP1 plays important roles in checkpoint signaling and repair for DNA double-strand breaks. We found that a colon cancer cell line, SW48, expressed a splicing variant form of 53BP1, which lacks the residues corresponding to exons 10 and 11. Activation of ATM and phosphorylation of ATM and ATR targets occurred in SW48 cells in response to X-irradiation, and these X-ray-induced responses were not enhanced by expression of full-length 53BP1 in SW48 cells, indicating that this splicing variant fully activates the major checkpoint signaling in SW48 cells. In contrast, the expression of full-length 53BP1 in SW48 cells promoted the repair of X-ray-induced DNA damage, evidenced by faster disappearance of X-ray-induced {gamma}-H2AX foci, a marker for DNA damage, and less residual chromosomal aberrations after X-irradiation. We conclude that the two major roles of 53BP1, the checkpoint signaling and repair for DNA damage, can be functionally separated.

60 APPLIED LIFE SCIENCES↗

Chk1 phosphorylation at Ser286 and Ser301 occurs with both stalled DNA replication and damage checkpoint stimulation

We previously reported Chk1 to be phosphorylated at Ser286 and Ser301 by cyclin-dependent kinase (Cdk) 1 during mitosis [T. Shiromizu et al., Genes Cells 11 (2006) 477-485]. Here, we demonstrated that Chk1-Ser286 and -Ser301 phosphorylation also occurs in hydroxyurea (HU)-treated or ultraviolet (UV)-irradiated cells. Unlike the mitosis case, however, Chk1 was phosphorylated not only at Ser286 and Ser301 but also at Ser317 and Ser345 in the checkpoint response. Treatment with Cdk inhibitors diminished Chk1 phosphorylation at Ser286 and Ser301 but not at Ser317 and Ser345 with the latter. In vitro analyses revealed Ser286 and Ser301 on Chk1 to serve as two major phosphorylation sites for Cdk2. Immunoprecipitation analyses further demonstrated that Ser286/Ser301 and Ser317/Ser345 phosphorylation occur in the same Chk1 molecule during the checkpoint response. In addition, Ser286/Ser301 phosphorylation by Cdk2 was observed in Chk1 mutated to Ala at Ser317 and Ser345 (S317A/S345A), as well as Ser317/Ser345 phosphorylation by ATR was in S286A/S301A. Therefore, Chk1 phosphorylation in the checkpoint response is regulated not only by ATR but also by Cdk2.

60 APPLIED LIFE SCIENCES↗

Silencing of poly(ADP-ribose) glycohydrolase sensitizes lung cancer cells to radiation through the abrogation of DNA damage checkpoint

Highlights: •Radiosensitization by PARG silencing was observed in multiple lung cancer cells. •PAR accumulation was enhanced by PARG silencing after DNA damage. •Radiation-induced G2/M arrest and checkpoint activation were impaired by PARG siRNA. -- Abstract: Poly(ADP-ribose) glycohydrolase (PARG) is a major enzyme that plays a role in the degradation of poly(ADP-ribose) (PAR). PARG deficiency reportedly sensitizes cells to the effects of radiation. In lung cancer, however, it has not been fully elucidated. Here, we investigated whether PARG siRNA contributes to an increased radiosensitivity using 8 lung cancer cell lines. Among them, the silencing of PARG induced a radiosensitizing effect in 5 cell lines. Radiation-induced G2/M arrest was largely suppressed by PARG siRNA in PC-14 and A427 cells, which exhibited significantly enhanced radiosensitivity in response to PARG knockdown. On the other hand, a similar effect was not observed in H520 cells, which did not exhibit a radiosensitizing effect. Consistent with a cell cycle analysis, radiation-induced checkpoint signals were not well activated in the PC-14 and A427 cells when treated with PARG siRNA. These results suggest that the increased sensitivity to radiation induced by PARG knockdown occurs through the abrogation of radiation-induced G2/M arrest and checkpoint activation in lung cancer cells. Our findings indicate that PARG could be a potential target for lung cancer treatments when used in combination with radiotherapy.

63 RADIATION, THERMAL, AND OTHER ENVIRON. POLLUTAN↗

Insulin-like growth factor I receptor regulates the radiation-induced G2/M checkpoint in HeLa cells

Highlights: • Inhibition of IGF-IR elongated radiation-induced G2 arrest in HeLa cells. • Inhibition of PI3K or AKT, but not MEK, induced the same effect. • IGF-IR inhibition did not essentially affect DSB repair kinetics. • Radiation-induced CHK1 nuclear localization lasted longer in IGF-IR-inhibited cells. • IGF-IR may regulate G2/M checkpoint in part via CHK1 shuttling within the cell. Insulin-like growth factor I receptor (IGF-IR) plays pivotal roles in various biological events, including cell growth, transformation, survival, and DNA repair. In this study, we explored its possible involvement in cell cycle checkpoints, using HeLa cells expressing the fluorescent ubiquitination-based cell cycle indicator (Fucci). We found that IGF-IR inhibitor delayed release from radiation-induced G2 arrest, as demonstrated by FACS and pedigree analysis of Fucci fluorescence. Elongated G2 arrest was also induced by inhibitors of phosphatidylinositol-3 kinase (PI3K) and AKT, but not by inhibitor of MEK, which are two major IGF-IR downstream signaling pathways. Double-strand break (DSB) repair kinetics were not affected by IGF-IR inhibitor. CHK1 inhibitor abrogated radiation-induced G2 arrest, whereas radiation-induced phosphorylation of CHK1 at Ser 345 or Ser 296 was decreased by the IGF-IR inhibitor. However, radiation-induced nuclear localization of CHK1 was prolonged in IGF-IR inhibitor–treated cells in comparison with cells that received radiation alone; in the latter, CHK1 returned to the original diffuse distribution in conjunction with release from G2 arrest. We conclude that IGF-IR directly regulates the G2/M checkpoint via the PI3K/AKT pathway without influencing DSB repair, in part by controlling CHK1 localization between the nucleus and cytoplasm.

60 APPLIED LIFE SCIENCES↗

Multicenter Evaluation of the Tolerability of Combined Treatment With PD-1 and CTLA-4 Immune Checkpoint Inhibitors and Palliative Radiation Therapy

Purpose: To analyze immune-related adverse events (ir-AEs) in patients treated with radiation and immune checkpoint blockade. Methods and Materials: We retrospectively reviewed records from patients with metastatic non-small cell lung cancer, melanoma, or renal cell cancer who received at least 1 cycle of a CTLA-4 or PD-1 inhibitor and radiation. Immune-related adverse events, defined using Common Terminology Criteria for Adverse Events version 4.0, were tabulated in relation to treatment variables, and associations with sequencing and timing were assessed. Results: We identified 133 patients, of whom 28 received a CTLA-4 inhibitor alone, 88 received a PD-1 inhibitor alone, and 17 received both classes of inhibitors either sequentially (n=13) or concurrently (n=4). Fifty-six patients received radiation within 14 days of an immune checkpoint inhibitor. Forty-six patients experienced at least 1 ir-AE (34.6%). Patients receiving both CTLA-4 and PD-1 inhibitors experienced more any-grade ir-AEs as compared with either individually (71% vs 29%, P=.0008). Any-grade ir-AEs occurred in 39% of patients in whom radiation was administered within 14 days of immunotherapy, compared with 23% of other patients (P=.06) and more often in patients who received higher equivalent dose in 2-Gy fractions (EQD2) EQD2 (P=.01). However, most toxicities were mild. There were no associations between site irradiated and specific ir-AEs. Conclusions: Our data suggest the combination of focal palliative radiation and CTLA-4 and/or PD-1 inhibitors is well tolerated, with manageable ir-AEs that did not seem to be associated with the particular site irradiated. Although conclusions are limited by the heterogeneity of patients and treatments, and future confirmatory studies are needed, this information can help guide clinical practice for patients receiving immune checkpoint therapy who require palliative radiation therapy.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Coordinate action of distinct sequence elements localizes checkpoint kinase Hsl1 to the septin collar at the bud neck in Saccharomyces cerevisiae

Passage through the eukaryotic cell cycle requires processes that are tightly regulated both spatially and temporally. Surveillance mechanisms (checkpoints) exert quality control and impose order on the timing and organization of downstream events by impeding cell cycle progression until the necessary components are available and undamaged and have acted in the proper sequence. In budding yeast, a checkpoint exists that does not allow timely execution of the G2/M transition unless and until a collar of septin filaments has properly assembled at the bud neck, which is the site where subsequent cytokinesis will occur. An essential component of this checkpoint is the large (1518-residue) protein kinase Hsl1, which localizes to the bud neck only if the septin collar has been correctly formed. Hsl1 reportedly interacts with particular septins; however, the precise molecular determinants in Hsl1 responsible for its recruitment to this cellular location during G2 have not been elucidated. We performed a comprehensive mutational dissection and accompanying image analysis to identify the sequence elements within Hsl1 responsible for its localization to the septins at the bud neck. Unexpectedly, we found that this targeting is multipartite. A segment of the central region of Hsl1 (residues 611–950), composed of two tandem, semiredundant but distinct septin-associating elements, is necessary and sufficient for binding to septin filaments both in vitro and in vivo. However, in addition to 611–950, efficient localization of Hsl1 to the septin collar in the cell obligatorily requires generalized targeting to the cytosolic face of the plasma membrane, a function normally provided by the C-terminal phosphatidylserine-binding KA1 domain (residues 1379–1518) in Hsl1 but that can be replaced by other, heterologous phosphatidylserine-binding sequences.

59 BASIC BIOLOGICAL SCIENCES↗

Optimal checkpointing of real-time tasks

Analytical models for the design and evaluation of checkpointing of real-time tasks are developed. First, the execution of a real-time task is modeled under a common assumption of perfect coverage of on-line detection mechanisms (which is termed a basic model). Then, the model is generalized (to an extended model) to include more realistic cases, i.e., imperfect coverages of on-line detection mechanisms and acceptance tests. Finally, an optimal placement of checkpoints is determined to minimize the mean task execution time while the probability of an unreliable result (or lack of confidence) is kept below a specified level. In the basic model, it is shown that equidistant intercheckpoint intervals are optimal, whereas this is not necessarily true in the extended model. An algorithm for calculating the optimal number of checkpoints and intercheckpoint intervals is presented with some numerical examples for the extended model.

Shin, Kang G.↗

Combining Partial Redundancy and Checkpointing for HPC

Today's largest High Performance Computing (HPC) systems exceed one Petaflops (10^15 floating point operations per second) and exascale systems are projected within seven years. But reliability is becoming one of the major challenges faced by exascale computing. With billion-core parallelism, the mean time to failure is projected to be in the range of minutes or hours instead of days. Failures are becoming the norm rather than the exception during execution of HPC applications. Current fault tolerance techniques in HPC focus on reactive ways to mitigate faults, namely via checkpoint and restart (C/R). Apart from storage overheads, C/R-based fault recovery comes at an additional cost in terms of application performance because normal execution is disrupted when checkpoints are taken. Studies have shown that applications running at a large scale spend more than 50\% of their total time saving checkpoints, restarting and redoing lost work. Redundancy is another fault tolerance technique, which employs redundant processes performing the same task. If a process fails, a replica of it can take over its execution. Thus, redundant copies can decrease the overall failure rate. The downside of redundancy is that extra resources are required and there is an additional overhead on communication and synchronization. This work contributes a model and analyzes the benefit of C/R in coordination with redundancy at different degrees to minimize the total wallclock time and resources utilization of HPC applications. We further conduct experiments with an implementation of redundancy within the MPI layer on a cluster. Our experimental results confirm the benefit of dual and triple redundancy - but not for partial redundancy - and show a close fit to the model. At 80,000 processes, dual redundancy requires twice the number of processing resources for an application but allows two jobs of 128 hours wallclock time to finish within the time of just one job without redundancy. For narrow ranges of processor counts, partial redundancy results in the lowest time. Once the count exceeds 770, 000, triple redundancy has the lowest overall cost. Thus, redundancy allows one to trade-off additional resource requirements against wallclock time, which provides a tuning knob for users to adapt to resource availabilities.

Elliott, James↗