DOE OSTI · 1567349
Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows, In: SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Abstract
As system scales and application complexity grow, managing and processing simulation data has become a significant challenge. While recent approaches based on data staging and in-situ/in-transit data processing are promising, dynamic data volumes and distributions, such as those occurring in AMR-based simulations, make the efficient use of these techniques challenging. In this paper we propose cross-layer adaptations that address these challenges and respond at runtime to dynamic data management requirements. Specifically we explore (1) adaptations of the spatial resolution at which the data is processed, (2) dynamic placement and scheduling of data processing kernels, and (3) dynamic allocation of in-transit resources. We also exploit coordinated approaches that dynamically combine these adaptations at the different layers. We evaluate the performance of our adaptive cross-layer management approach on the Intrepid IBM-BlueGene/P and Titan Cray-XK7 systems using Chombo-based AMR applications, and demonstrate its effectiveness in improving overall time-to-solution and increasing resource efficiency.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Jin, Tong, Zhang, Fan, Sun, Qian, Bui, Hoang, Parashar, Manish, Yu, Hongfeng, Klasky, Scott, Podhorszki, Norbert, Abbasi, Hasan. 2013-01-01. Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows, In: SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. https://doi.org/10.1145/2503210.2503301. https://searchnasa.com/record/osti%3A1567349/using-cross-layer-adaptations-for-dynamic-data-management-in-large-scale-coupled-scientific-workflow
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Search NASA by Indiciae: BibTeX · RIS.