Crunching Large Graphs with Commodity Processors

Abstract

Crunching large graphs is the basis of many emerging appli- cations, such as social network analysis and bioinformatics. Graph analytics algorithms exhibit little locality and therefore present significant performance challenges. Hardware multi- threading systems (e.g, Cray XMT) show that with enough concurrency, we can tolerate long latencies. Unfortunately, this solution is not available with commodity parts. Our goal is to develop a latency-tolerant system built out of commodity parts and mostly in software. The proposed system includes a runtime that supports a large number of lightweight contexts, full-bit synchronization and a memory manager that provides a high-latency but high-bandwidth global shared memory. This paper lays out the vision for our system, and justifies its feasibility with a performance analysis of the run- time for latency tolerance.