A Scalable Distributed Parallel Breadth-First Search Algorithm on BlueGene/L

Many emerging large-scale data science applications require searching large graphs distributed across multiple memories and processors. This paper presents a distributed breadth- first search (BFS) scheme that scales for random graphs with up to three billion vertices and 30 billion edges. Scalabili...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Proceedings of the 2005 ACM/IEEE conference on Supercomputing S. 25
Hauptverfasser: Yoo, Andy, Chow, Edmond, Henderson, Keith, McLendon, William, Hendrickson, Bruce, Catalyurek, Umit
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: Washington, DC, USA IEEE Computer Society 2005
IEEE
Schriftenreihe:ACM Conferences
Schlagworte:
Area measurement
Computer systems organization > Architectures > Distributed architectures > Grid computing
Computer systems organization > Architectures > Parallel architectures > Multicore architectures
Computer systems organization > Architectures > Serial architectures > Superscalar architectures
Computing methodologies > Artificial intelligence > Search methodologies
Computing methodologies > Artificial intelligence > Search methodologies > Discrete space search
Computing methodologies > Artificial intelligence > Search methodologies > Game tree search
Computing methodologies > Modeling and simulation > Simulation types and techniques > Massively parallel and high-performance simulations
Extraterrestrial measurements
Gain measurement
Government
Laboratories
Large-scale systems
Partitioning algorithms
Research and development
Scalability
Software and its engineering > Software organization and properties > Software system structures > Distributed systems organizing principles > Grid computing
Testing
Theory of computation > Design and analysis of algorithms > Data structures design and analysis > Sorting and searching
ISBN:1595930612, 9781595930613
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Many emerging large-scale data science applications require searching large graphs distributed across multiple memories and processors. This paper presents a distributed breadth- first search (BFS) scheme that scales for random graphs with up to three billion vertices and 30 billion edges. Scalability was tested on IBM BlueGene/L with 32,768 nodes at the Lawrence Livermore National Laboratory. Scalability was obtained through a series of optimizations, in particular, those that ensure scalable use of memory. We use 2D (edge) partitioning of the graph instead of conventional 1D (vertex) partitioning to reduce communication overhead. For Poisson random graphs, we show that the expected size of the messages is scalable for both 2D and 1D partitionings. Finally, we have developed efficient collective communication functions for the 3D torus architecture of BlueGene/L that also take advantage of the structure in the problem. The performance and characteristics of the algorithm are measured and reported.
Bibliographie:SourceType-Conference Papers & Proceedings-1
ObjectType-Conference Paper-1
content type line 25
ISBN:1595930612
9781595930613
DOI:10.1109/SC.2005.4