DOE OSTI · 1842617
WIRE: Resource-efficient Scaling with Online Prediction for DAG-based Workflows
Abstract
This paper introduces WIRE that manages resources for the DAG-based workflows on IaaS clouds. WIRE predicts and plans resources over the MAPE (Monitor-Analyze-Plan-Execute) loops to: 1) Estimate task performance with online data, 2) Conduct simulations to predict the upcoming loads based on online estimates and workflow DAGs, 3) Apply a resource-steering policy to size cloud instance pools for the maximal parallelism that is consistent with low cost. We implement WIRE on Pegasus WMS/HTCondor and evaluate its performance on the ExoGENI network cloud. The results show that WIRE attains low resource cost with the performance that is typically within a factor of two of optimal.
Keep this discovery
Explore connections, maps & timelines
Xie, Bing, Cao, Qiang, Kunjir, Mayuresh, Wan, Linli, Chase, Jeffrey, Mandal, Anirban, Rynge, Mats. 2021-10-01. WIRE: Resource-efficient Scaling with Online Prediction for DAG-based Workflows. https://doi.org/10.1109/cluster48925.2021.00025
Cite the original work for its findings. Save a collection to share your selection of sources.