DOE OSTI · 1843691
Elastic distributed training with fast convergence and efficient resource utilization
Abstract
Distributed learning is now routinely conducted on cloud as well as dedicated clusters. Training with elastic resources brings new challenges and design choices. Prior studies focus on runtime performance and assume a static algorithmic behavior. In this work, by analyzing the impact of of resource scaling on convergence, we introduce schedules for synchronous stochastic gradient descent that proactively adapt the number of learners to reduce training time and improve convergence. Our approach no longer assumes a constant number of processors throughout training. In our experiment, distributed stochastic gradient descent with dynamic schedules and reduction momentum achieves better convergence and significant speedups over prior static ones. Numerous distributed training jobs running on cloud may benefit from our approach.
Keep this discovery
Explore connections, maps & timelines
Cong, Guojing. 2021-12-01. Elastic distributed training with fast convergence and efficient resource utilization. https://doi.org/10.1109/icmla52953.2021.00160. https://searchnasa.com/record/osti%3A1843691/elastic-distributed-training-with-fast-convergence-and-efficient-resource-utilization
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Search NASA by Indiciae: BibTeX · RIS.