CuSP: A Customizable Streaming Edge Partitioner for Distributed Graph Analytics

Graph analytics systems must analyze graphs with billions of vertices and edges which require several terabytes of storage. Distributed-memory clusters are often used for analyzing such large graphs since the main memory of a single machine is usually restricted to a few hundreds of gigabytes. This...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Proceedings - IEEE International Parallel and Distributed Processing Symposium s. 439 - 450
Hlavní autoři: Hoang, Loc, Dathathri, Roshan, Gill, Gurbinder, Pingali, Keshav
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 01.05.2019
Témata:
Clustering algorithms
distributed-memory computing
graph analytics
graph partitioning
Heuristic algorithms
Microsoft Windows
Mirrors
Parallel processing
Partitioning algorithms
streaming partitioners
Two dimensional displays
ISSN:1530-2075
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Graph analytics systems must analyze graphs with billions of vertices and edges which require several terabytes of storage. Distributed-memory clusters are often used for analyzing such large graphs since the main memory of a single machine is usually restricted to a few hundreds of gigabytes. This requires partitioning the graph among the machines in the cluster. Existing graph analytics systems usually come with a built-in partitioner that incorporates a particular partitioning policy, but the best partitioning policy is dependent on the algorithm, input graph, and platform. Therefore, built-in partitioners are not sufficiently flexible. Stand-alone graph partitioners are available, but they too implement only a small number of partitioning policies. This paper presents CuSP, a fast streaming edge partitioning framework which permits users to specify the desired partitioning policy at a high level of abstraction and generates high-quality graph partitions fast. For example, it can partition wdc12, the largest publicly available web-crawl graph, with 4 billion vertices and 129 billion edges, in under 2 minutes for clusters with 128 machines. Our experiments show that it can produce quality partitions 6× faster on average than the state-of-the-art standalone partitioner in the literature while supporting a wider range of partitioning policies.
ISSN:1530-2075
DOI:10.1109/IPDPS.2019.00054