Search NASA⌕ Search

SEARCH · Search NASA

Results for “restart”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
NASA NTRS2003

Software for Refining or Coarsening Computational Grids

A computer program performs calculations for refinement or coarsening of computational grids of the type called structured (signifying that they are geometrically regular and/or are specified by relatively simple algebraic expressions). This program is designed to facilitate analysis of the numerical effects of changing structured grids utilized in computational fluid dynamics (CFD) software. Unlike prior grid-refinement and -coarsening programs, this program is not limited to doubling or halving: the user can specify any refinement or coarsening ratio, which can have a noninteger value. In addition to this ratio, the program accepts, as input, a grid file and the associated restart file, which is basically a file containing the most recent iteration of flow-field variables computed on the grid. The program then refines or coarsens the grid as specified, while maintaining the geometry and the stretching characteristics of the original grid. The program can interpolate from the input restart file to create a restart file for the refined or coarsened grid. The program provides a graphical user interface that facilitates the entry of input data for the grid-generation and restart-interpolation routines.

Daines, Russell↗
NASA NTRS2020

Building the RS-25 Engine for NASA’s Next Generation of Exploration.

NASA is aggressively pursuing a human lunar return to the Moon with the Artemis Program. The Space Launch System (SLS) is critical to the transportation architecture, providing both crew and cargo capability. To accelerate SLS development, NASA settled on space shuttle heritage propulsion technologies. Early missions will use repurposed RS-25 engines from the Shuttle Program adapted to SLS requirements. For future missions, however, NASA and engine contractor Aerojet Rocketdyne are restarting RS-25 production with a goal of using the latest manufacturing technologies to produce an RS-25 variant that will cost at least 30 percent less than the shuttle-era engines developed and flown for almost 30 years. The prospect of restarting the production line on these engines after many years and modifying the engine to reduce cost and better match the needs of the SLS vehicle presented unique challenges and opportunities for NASA and Aerojet Rocketdyne. NASA and AR are now deep into the process of adapting 16 RS-25 engines for the first four SLS Artemis flights and building the first RS-25 “Restart” engines for future Artemis missions. In addition to navigating programmatic, technical, and logistical challenges with the current RS-25 production work, NASA and AR have partnered to pursue new methods of building this engine through advanced manufacturing techniques to further reduce the cost and schedule required to build each RS-25 engine to contribute to the long-term affordability of the SLS vehicle. This presentation will discuss the goals for the restart program, the challenges, and results to date.

Jessica Jean Wood↗
DOE OSTI2020

Developing Separate Effects Transient Test Experiments Using an Out-of-Pile Flowing Water Loop

In 2017 the Transient Reactor Test Facility was restarted after being placed in a standby state since 1994. This Test Reactor’s restart has since enabled the progressive development of new nuclear technologies within the U.S. that were previously required to outsource to other countries. While the reactor’s restart was a large feat worthy of recognition, the experimental use of its characteristics has required the further development of in-pile experimental infrastructure sufficient to support programmatic needs. This hardware have taken the form of capsule designs (compact and elongated) as well as loop concepts representing the phenomena of interest for a subset of separate effects tests desire for each respective testing campaign. The Transient Testing Program has a large-integrated effort that aligns with the U.S. Department of Energy’s current needs. This study compliments those programmatic elements by developing, fabricating and demonstrating a full-scale flowing water loop in an out-of-pile environment. The goal of this effort is to develop a pragmatic understanding of the engineering capabilities and limitations associated with geometric form-factors, metering technology, and controls logic under representative thermal hydraulic conditions that would be experienced within the TREAT reactor during an in-pile reactivity initiated accident test. The outcomes of this study result in an evaluation of the conceptual design of a comprehensive flowing water loop including objective figures of merit (FoMs) for comparing unique instrumentation and the basis for their selection during operations. These efforts directly contribute to and are required for the further advancement of transient testing capabilities within the U.S.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2026

Model Data Archive Associated with Manuscript "Fire-altered Carbon Pools Create Disturbance Memory in Stream Dissolved Organic Carbon"

This data package supports the publication “Fire-altered Carbon Pools Create Disturbance Memory in Stream Dissolved Organic Carbon” by Li et al. (2026). The package contains processed model inputs, configuration files, restart files, simulation outputs, scripts, and visualization products used to evaluate post-fire dissolved organic carbon (DOC) dynamics in the Naches River Watershed, Washington, USA, following the 2021 Schneider Springs Fire. The modeling workflow couples ELM-BGC, the biogeochemistry-enabled Energy Exascale Earth System Model Land Model; ATS, the Advanced Terrestrial Simulator for integrated surface-subsurface hydrology; and PFLOTRAN, a reactive transport model for multicomponent aqueous geochemistry. Together, these models simulate how wildfire-induced changes in vegetation, litter, coarse woody debris, and soil organic matter influence DOC production, transport, and reaction from burned hillslopes to stream networks. The archive includes preprocessed meteorological, geospatial, hydrologic, and biogeochemical forcing data; ELM-BGC-derived DOC source terms; ATS mesh files; PFLOTRAN reactive-transport inputs; model configuration files; spin-up and transient restart files; watershed-scale diagnostic outputs; stream concentration time series; and figures or visualization files used to inspect and reproduce key results. File types include Hierarchical Data Format 5 (HDF5) files for gridded forcing and model-coupling data, model input and configuration files for ELM-BGC, ATS, and PFLOTRAN, restart and simulation-output files generated by the modeling workflow, tabular or time-series diagnostic outputs, scripts for post-processing and figure generation, and image or visualization products associated with the manuscript. Use of the package depends on the intended task. Re-running the simulations requires the relevant modeling software, including ELM-BGC, ATS, and PFLOTRAN as ATS's geochemical engine. Inspecting outputs and reproducing figures requires Python with scientific plotting libraries such as Matplotlib, and three-dimensional model outputs may be viewed with ParaView. Geographic information system files or maps may be inspected with ArcGIS Pro or comparable GIS software. The data package is intended to enable traceability, reuse, and partial reproduction of the coupled land-to-watershed hydro-biogeochemical modeling workflow used to test how wildfire disturbance affects terrestrial carbon pools and downstream DOC dynamics.

ATS↗
DOE OSTI2020

Implementing Software Resiliency in HPX for Extreme Scale Computing

The DOE Office of Science Exascale Computing Project (ECP) outlines the next milestones in the supercomputing domain. The target computing systems under the project will deliver 10x performance while keeping the power budget under 30 megawatts. With such large machines, the need to make applications resilient has become paramount. The benefits of adding resiliency to mission critical and scientific applications, includes the reduced cost of restarting the failed simulation both in terms of time and power. Most of the current implementation of resiliency at the software level makes use of a Coordinated Checkpoint and Restart (C/R). This technique of resiliency generates a consistent global snapshot, also called a checkpoint. Generating snapshots involves global communication and coordination and is achieved by synchronizing all running processes. The generated checkpoint is then stored in some form of persistent storage. On failure detection, the runtime initiates a global rollback to the most recent previously saved checkpoint. This involves aborting all running processes, rolling them back to the previous state and restarting them.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

IM3/HyperFACETS Thermodynamic Global Warming (TGW) Simulation Datasets

Publication For a thorough description of the methods, see the peer-reviewed paper: Jones, A.D., Rastogi, D., Vahmani, P. et al. Continental United States climate projections based on thermodynamic modification of historical weather. Sci Data 10, 664 (2023). https://doi.org/10.1038/s41597-023-02485-5 Overview The IM3 / HyperFACETS climate simulations provide 40-year historical (1980-2019) as well as four 80-year future simulations (2020-2099) over the U.S. The future simulations are split into near (2020-2059) and far future (2060-2099) segments. The future scenarios span a range of plausible changes in future climate (both Global Circulation Model (GCM) and Representative Concentration Pathways/Shared Socioeconomic Pathway (RCP/SSP) dimensions). The simulations provide climate variables with high spatiotemporal resolution (25 hourly variables and 207 3-hourly variables at 12 km2). The datasets are generated using dynamical downscaling with the WRF (Weather Research and Forecasting) model (version 4.2.1) and therefore preserve physical consistency across variables. WRF is a state-of-the-art, fully compressible, non-hydrostatic, mesoscale numerical weather prediction model. WRF is coupled with an urban canopy model (UCM), which resolves urban surfaces. The future scenarios were developed using a thermodynamic global warming approach where past events are replayed under a range of future warming conditions. These scenarios therefore provide a perspective on potential increases in extreme event intensity, geographic scope, and duration, with previously non-extreme conditions potentially crossing new thresholds to be considered extreme by today's standards. This approach is not intended to estimate future changes in extreme event frequency that might result from changes in large-scale atmospheric dynamics. This dataset has NOT been bias corrected. A bias corrected version of selected variables is under development and will be released here when available. Scenarios Files Data for each scenario is provided in weekly NetCDF files. 25 variables are available at hourly resolution, and 207 variables are available at three-hourly resolution. Spatial resolution is 12km and spans the conterminous United States (CONUS), including some areas of Canada and Mexico, resulting in a grid of 424 by 299 cells. The spatial projection is a Lambert Conformal Conic with the following proj-string: "+proj=lcc +lat_0=40.0000076293945 +lon_0=-97 +lat_1=30 +lat_2=45 +x_0=0 +y_0=0 +R=6370000 +units=m +no_defs". The available scenarios and simulation periods are listed below: historical | 1980 - 2019 rcp45cooler | 2020 - 2059 rcp45cooler | 2060 - 2099 rcp45hotter | 2020 - 2059 rcp45hotter | 2060 - 2099 rcp85cooler | 2020 - 2059 rcp85cooler | 2060 - 2099 rcp85hotter | 2020 - 2059 rcp85hotter | 2060 - 2099 * The first year (1979, 2019, and 2059) of data within each scenario represents a model warmup period and should not be used. These are located in the `spinup_files` directory. Historical year 2020 is considered an extra year of data beyond the simulation period and can be found in the `additional_files` directory. For information on specific variables and a more in-depth discussion of methodology, please refer to the data landing page at https://tgw-data.msdlive.org. Delta Warming Files The global and CONUS warming deltas for each scenario are provided in degrees Celsius annually and monthly. Restart Files Yearly restart files are provided for each scenario which can be used to restart the WRF model at a particular point in time. Spinup Files The first year of data within each simulation period represents a model warmup period and should not be used. The files are provided here for the sake of reproducibility. Additional Files Additional years of data are provided as an extension of the historic simulation.

Jones, Andrew D.↗
NASA NTRS1985

Theoretical and software considerations for general dynamic analysis using multilevel substructured models

The dynamic analysis of complex structural systems using the finite element method and multilevel substructured models is presented. The fixed-interface method is selected for substructure reduction because of its efficiency, accuracy, and adaptability to restart and reanalysis. This method is extended to reduction of substructures which are themselves composed of reduced substructures. The implementation and performance of the method in a general purpose software system is emphasized. Solution algorithms consistent with the chosen data structures are presented. It is demonstrated that successful finite element software requires the use of software executives to supplement the algorithmic language. The complexity of the implementation of restart and reanalysis porcedures illustrates the need for executive systems to support the noncomputational aspects of the software. It is shown that significant computational efficiencies can be achieved through proper use of substructuring and reduction technbiques without sacrificing solution accuracy. The restart and reanalysis capabilities and the flexible procedures for multilevel substructured modeling gives economical yet accurate analyses of complex structural systems.

Schmidt, R. J.↗
NASA NTRS1991

Visualization of the freeze/thaw characteristics of a copper/water heat pipe - Effects of non-condensible gas

The freeze/thaw characteristics of a copper/water heat pipe of rectangular cross section were investigated experimentally to determine the effect of variations in the amount of non-condensible gases (NCG) present. The transient internal temperature profiles in both the liquid and vapor channels are presented along with contours of the frozen fluid configuration obtained through visual observation. Several interesting phenomena were observed including total blockage of the vapor channel by a solid plug, evaporator dryout during restart, and freezing blowby. In addition, the restart characteristics are shown to be strongly dependent upon the shutdown procedure used prior to freezing, indicating that accurate prediction of the startup or restart characteristics requires a complete thermal history. Finally, the experimental results indicate that the freeze/thaw characteristics of room temperature heat pipes may be significantly different from those occurring in higher temperature, liquid metal heat pipes due to differences in the vapor pressures in the frozen condition.

Ochterbeck, J. M.↗
NASA NTRS1992

The Saturn V F-1 engine revisited

This paper describes how the Saturn V F-1 engine could be resurrected for potential use in a new heavy lift launch vehicle (HLLV). Performance, history, and reliability of the F-1 will be discussed and a 1990s F-1 engine configuration presented. The status of residual hardware, production capability, test-stand availability, and a general program plan to accomplish this restart will be presented. Similarities between restarting the F-1 program and restarting the Atlas and Delta engine programs will be discussed.

Shelton, B. W.↗
NASA NTRS2012

Near Earth Asteroid Human Mission Possibilities Using Nuclear Thermal Rocket (NTR) Propulsion

The NTR is a proven technology that generates high thrust and has a specific impulse (Isp (is) approximately 900 s) twice that of today's best chemical rockets. During the Rover and NERVA (Nuclear Engine for Rocket Vehicle Applications) programs, twenty rocket reactors were designed, built and ground tested. These tests demonstrated: (1) a wide range of thrust; (2) high temperature carbide-based nuclear fuel; (3) sustained engine operation; (4) accumulated lifetime; and (5) restart capability - all the requirements needed for a human mission to Mars. Ceramic metal fuel was also evaluated as a backup option. In NASA's recent Mars Design reference Architecture (DRA) 5.0 study, the NTR was selected as the preferred propulsion option because of its proven technology, higher performance, lower launch mass, versatile vehicle design, simple assembly, and growth potential. In contrast to other advanced propulsion options, NTP requires no large technology scale-ups. In fact, the smallest engine tested during the Rover program - the 25 klbf 'Pewee' engine is sufficient for a human Mars mission when used in a clustered engine configuration. The 'Copernicus crewed NTR Mars transfer vehicle design developed for DRA 5.0 has significant capability that can enable reusable '1-year' round trip human missions to candidate near Earth asteroids (NEAs) like 1991 JW in 2027, or 2000 SG344 and Apophis in 2028. A robotic precursor mission to 2000 SG344 in late 2023 could provide an attractive Flight Technology Demonstration of a small NTR engine that is scalable to the 25 klbf-class engine used for human missions 5 years later. In addition to the detailed scientific data gathered from on-site inspection, human NEA missions would also provide a valuable 'check out' function for key elements of the NTR transfer vehicle (its propulsion module, TransHab and life support systems, etc.) in a 'deep space' environment prior to undertaking the longer duration Mars orbital and landing missions that would follow. The initial mass in low Earth orbit required for a mission to Apophis is approximately 323 t consisting of the NTR propulsion module ((is) approximately 138 t), the integrated saddle truss and LH2 drop tank assembly ((is) approximately 123 t), and the 6-crew payload element ((is) approximately 62 t). The later includes a multi-mission Space Excursion Vehicle (MMSEV) used for close-up examination and sample gathering. The total burn time and required restarts on the three 25 klbf 'Pewee-class' engines operating at Isp (is) approximately 906 s, are approximately 76.2 minutes and 4, respectively, well below the 2 hours and 27 restarts demonstrated on the NERVA eXperimental Engine, the NRX-XE. The paper examines the benefits, requirements and characteristics of using NTP for the above NEA missions. The impacts on vehicle design of HLV payload volume and lift capability, crew size, and reusability are also quantified.

Borowski, Stanley↗
DOE OSTI2020

Checkpoint triggering in a computer system

According to an aspect, a method for triggering creation of a checkpoint in a computer system includes executing a task in a processing node of the computer system. A monitoring block size is determined for the checkpoint. A checkpoint interval is determined based on the monitoring block size, a checkpoint bandwidth, and a failure rate of the computer system. Based on determining that the checkpoint interval has elapsed, the checkpoint including state data of the task is created to enable restarting execution of the task upon a restart operation. The state data of the checkpoint is restored from a memory responsive to detecting an error condition at the processing node. Execution of the task is restarted in the processing node based on the state data restored from the memory.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Perspective on the oil-dominated gas hydrate plugging conceptual picture as applied to transient Shut-In/Restart

Gas hydrate formation is rarely an issue during continuous production conditions. Unplanned and/or extended shut-downs comprise the majority of scenarios where overlapping concern and uncertainty for gas hydrate blockages exist. To date, most experimental studies and theoretical analysis of gas hydrate formation derive from the steady-state conceptual picture. Yet, significant distinctions induced by phase separation and prolonged cooling suggest an altered conceptual picture is required for transient shut-in/restart operations. To address this outstanding issue, this work presents a conceptual picture for the base case of a non-surface active hydrocarbon undergoing cold restart without enacted mitigation efforts. First, phase separation upon shut-in leads to the localization of gas hydrate ingredients due to fluids density differences and pipeline inclination. Cooling to the seafloor temperature, then, induces gas hydrate film formation at the hydrocarbon interface and maximizes subcooling, resulting in rapid and confined gas hydrate growth occurring upon restart. Furthermore, this ultimately leads to plugging dominated by the deposition of porous, low conversion gas hydrate masses. Direct visual observations and results from Raman spectroscopy, hydrate contact angle and film growth, micromechanical force apparatus, rocking cell, and flow loop studies support the mechanisms outlined within each step. Similar to the influence provided by the original Colorado School of Mines steady-state conceptual picture that was introduced about 15 years ago, this transient counterpart builds a foundation that could inspire new directions of hydrate research and analysis in areas where uncertainty abounds.

02 PETROLEUM↗
DOE OSTI2021

Enhanced Data Efficiency Using Deep Neural Networks and Gaussian Processes for Aerodynamic Design Optimization

Adjoint-based optimization methods are attractive for aerodynamic shape design primarily due to their computational costs being independent of the dimensionality of the input space and their ability to generate high-fidelity gradients that can then be used in a gradient-based optimizer. This makes them very well suited for high-fidelity simulation based aerodynamic shape optimization of highly parametrized geometries such as aircraft wings. However, the development of adjoint-based solvers involve careful mathematical treatment and their implementation require detailed software development. Furthermore, they can become prohibitively expensive when multiple optimization problems are being solved, each requiring multiple restarts to circumvent local optima. In this work, we propose a machine learning enabled, surrogate-based framework that replaces the expensive adjoint solver, without compromising on predicting predictive accuracy. Specifically, we first train a deep neural network (DNN) from training data generated from evaluating the high-fidelity simulation model on a model-agnostic design of experiments on the geometry shape parameters. The optimum shape may then be computed by using a gradient-based optimizer coupled with the trained DNN. Subsequently, we also perform a gradient-free Bayesian optimization, where the trained DNN is used as the prior mean. We observe that the latter framework (DNN-BO) improves upon the DNN-only based optimization strategy for the same computational cost. Overall, this framework predicts the true optimum with very high accuracy, while requiring far fewer high-fidelity function calls compared to the adjoint-based method. Furthermore, we show that multiple optimization problems can be solved with the same machine learning model with high accuracy, to amortize the offline costs associated with constructing our models. Our methodology finds applications in the early stages of aerospace design. (C) 2021 Published by Elsevier Masson SAS.

Renganathan, S. Ashwin↗
DOE OSTI2023

Sensitivity Analysis of Transient Critical Heat Flux by RIA Under High-Pressure Flow Boiling Conditions in TRTL

A reactivity-initiated accident (RIA) is a design-basis accident under which critical heat flux (CHF) is likely to be exceeded. The operational margin for RIAs is currently determined using steady-state CHF lookup tables, which provide conservative estimates relative to transient CHF phenomena. The Transient Reactor Test Loop (TRTL) facility at Oregon State University is capable of performing out-of-pile rapid heating experiments representative of a RIA at conditions representative of a pressurized water reactor (PWR). Here, to further our understanding of and ability to predict transient CHF under PWR conditions, we performed a sensitivity analysis on a RELAP5-3D model of the TRTL facility coupled to the RAVEN code framework to define a proposed experimental test matrix to be performed at the TRTL facility. We then implemented a flow boiling CHF correlation into RELAP5-3D and performed a secondary sensitivity analysis inspecting the impact of the built-in RELAP5-3D CHF and heat transfer multipliers on both the prediction of CHF and key safety parameters, such as peak cladding temperature and heat flux. The results show that the multiplier with the highest influence toward the prediction of CHF occurrence and the safety parameters is the transient CHF multiplier. Operational performance envelopes have been developed for each of the test matrix cases and will be used for validation once the experiments are performed. The TRTL facility is currently performing shakedown testing to verify system performance prior to proceeding with the experimental campaign. Restart testing results include pump curve restart testing, pressure tests, and heater rod thermocouple transients.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2024

Tensor decompositions for count data that leverage stochastic and deterministic optimization

There is growing interest to extend low-rank matrix decompositions to multi-way arrays, or tensors. One fundamental low-rank tensor decomposition is the canonical polyadic decomposition (CPD). The challenge of fitting a low-rank, nonnegative CPD model to Poisson-distributed count data is of particular interest. Several popular algorithms use local search methods to approximate the maximum likelihood estimator (MLE) of the Poisson CPD model. Here, this work presents two new algorithms that extend state-of-the-art local methods for Poisson CPD. Hybrid GCP-CPAPR combines Generalized Canonical Decomposition (GCP) with stochastic optimization and CP Alternating Poisson Regression (CPAPR), a deterministic algorithm, to increase the probability of converging to the MLE over either method used alone. Restarted CPAPR with SVDrop uses a heuristic based on the singular values of the CPD model unfoldings to identify convergence toward optimizers that are not the MLE and restarts within the feasible domain of the optimization problem, thus reducing overall computational cost when using a multi-start strategy. We provide empirical evidence that indicates our approaches outperform existing methods with respect to converging to the Poisson CPD MLE.

CPAPR↗
DOE OSTI2023

Young but fading radio sources: searching for remnants among compact steep-spectrum radio sources

The incidence of young but fading radio sources provides important information on the life cycle of radio emission in radio-loud active galactic nuclei. Despite its importance for constraining the models of radio source evolution, there are no systematic studies of remnants in complete samples of young radio sources. We report results of the study of 18 compact steep-spectrum (CSS) radio sources, selected from the statistically complete B3-VLA (Very Large Array) CSS sample, characterized by a steep optically thin spectrum (α ≥ 1.0) and no core detection in earlier studies. Our deep multifrequency VLA, pc-scale Very Long Baseline Array, and enhanced Multi Element Remotely Linked Interferometer Network (e-MERLIN) observations allowed us to locate the core component in 10 objects. In three CSS sources, there is no clear evidence of present-time active regions, suggesting that they are likely in a remnant phase. Among sources with core detection, we find three objects that have no clear active regions (hotspots) at the edges of the radio structure, suggesting that the radio emission may have just restarted. Our results support a power-law distribution of the source ages, although the poor statistics prevents us from setting solid constraints on the percentage of remnants and restarted sources in subpopulations of radio sources.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

BBOS: Efficient HPC Storage Management via Burst Buffer Over-Subscription

To avoid access to PFS, dedicated BB allocation is preferred despite of severe BB underutilization. Recently, new all-flash HPC storage systems with integrated BB and PFS are proposed, which speed up access to PFS. For this reason, we adopt BB over-subscription allocation method by allowing HPC applications to use BB only for I/O phase for improving BB utilization. Unfortunately, BB over-subscription aggravates I/O interference and demotion overhead from BB to PFS, resulting in degraded performance. To minimize the performance degradation, we develop an I/O scheduler to prevent I/O congestion and a new transparent data management system based on checkpoint/restart characteristics of HPC applications. With the proposed approach, not only the BB utilization can be improved, but also high performance of applications is achieved. In our experiments, we find that BB utilization is improved at least 2.2x, and more stable and higher checkpoint performance is guaranteed compared to other approaches. Besides, we achieve up to 96.4% hit ratio of restart requests on BB and up to 3.1x higher restart performance than others.

Sung, H↗
DOE OSTI2022

Towards Low-Overhead Resilience for Data Parallel Deep Learning

Data parallel techniques have been widely adopted both in academia and industry as a tool to enable scalable training of deep learning models. At scale, DL training jobs can fail due to software or hardware bugs, may need to be preempted or terminated due to unexpected events, or may perform suboptimally because they were misconfigured. Under such circumstances, there is a need to recover and/or reconfigure data-parallel DL training jobs on-the-fly, while minimizing the impact on the accuracy of the DNN model and the runtime overhead. In this regard, state-of-art techniques adopted by the HPC community mostly rely on checkpoint-restart, which inevitably leads to loss of progress, thus increasing the runtime overhead. In this paper we explore alternative techniques that exploit the properties of modern deep learning frameworks (overlapping of gradient averaging and weight updates with local gradient computations through pipeline parallelism) to reduce the overhead of resilience/elasticity. To this end we introduce a failure simulation framework and two resilience strategies (immediate mini-batch rollback and lossy forward recovery), which we study compared with checkpoint-restart approaches in a variety of settings in order to understand the trade-offs between the accuracy loss of the DNN model and the runtime overhead.

data-parallel training↗