A Scalable Distributed Parallel Breadth-First Search Algorithm on BlueGene/L
Many emerging large-scale data science applications require searching large graphs distributed across multiple memories and processors. This paper presents a distributed breadth- first search (BFS) scheme that scales for random graphs with up to three billion vertices and 30 billion edges. Scalabili...
Saved in:
| Published in: | Proceedings of the 2005 ACM/IEEE conference on Supercomputing p. 25 |
|---|---|
| Main Authors: | , , , , , |
| Format: | Conference Proceeding |
| Language: | English |
| Published: |
Washington, DC, USA
IEEE Computer Society
2005
IEEE |
| Series: | ACM Conferences |
| Subjects: | |
| ISBN: | 1595930612, 9781595930613 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Many emerging large-scale data science applications require searching large graphs distributed across multiple memories and processors. This paper presents a distributed breadth- first search (BFS) scheme that scales for random graphs with up to three billion vertices and 30 billion edges. Scalability was tested on IBM BlueGene/L with 32,768 nodes at the Lawrence Livermore National Laboratory. Scalability was obtained through a series of optimizations, in particular, those that ensure scalable use of memory. We use 2D (edge) partitioning of the graph instead of conventional 1D (vertex) partitioning to reduce communication overhead. For Poisson random graphs, we show that the expected size of the messages is scalable for both 2D and 1D partitionings. Finally, we have developed efficient collective communication functions for the 3D torus architecture of BlueGene/L that also take advantage of the structure in the problem. The performance and characteristics of the algorithm are measured and reported. |
|---|---|
| AbstractList | Many emerging large-scale data science applications require searching large graphs distributed across multiple memories and processors. This paper presents a distributed breadth- first search (BFS) scheme that scales for random graphs with up to three billion vertices and 30 billion edges. Scalability was tested on IBM BlueGene/L with 32,768 nodes at the Lawrence Livermore National Laboratory. Scalability was obtained through a series of optimizations, in particular, those that ensure scalable use of memory. We use 2D (edge) partitioning of the graph instead of conventional 1D (vertex) partitioning to reduce communication overhead. For Poisson random graphs, we show that the expected size of the messages is scalable for both 2D and 1D partitionings. Finally, we have developed efficient collective communication functions for the 3D torus architecture of BlueGene/L that also take advantage of the structure in the problem. The performance and characteristics of the algorithm are measured and reported. |
| Author | Hendrickson, Bruce Catalyurek, Umit Chow, Edmond Henderson, Keith McLendon, William Yoo, Andy |
| Author_xml | – sequence: 1 givenname: Andy surname: Yoo fullname: Yoo, Andy organization: Lawrence Livermore National Laboratory, Livermore – sequence: 2 givenname: Edmond surname: Chow fullname: Chow, Edmond organization: D. E. Shaw Research and Development, New York – sequence: 3 givenname: Keith surname: Henderson fullname: Henderson, Keith organization: Lawrence Livermore National Laboratory, Livermore – sequence: 4 givenname: William surname: McLendon fullname: McLendon, William organization: Sandia National Laboratories, Albuquerque, NM – sequence: 5 givenname: Bruce surname: Hendrickson fullname: Hendrickson, Bruce organization: Sandia National Laboratories, Albuquerque, NM – sequence: 6 givenname: Umit surname: Catalyurek fullname: Catalyurek, Umit organization: Ohio State University, Columbus |
| BookMark | eNqFkLFOwzAURS0BErR0YGaxGBBLWr84TuKxDbQgRQIpMFt28kINblLsZODvoSo70x3O0RnuhJx2fYeEXAGbAzC5qIp5zJiYJydkAkIKyVkK8TmZhfDBGAOZShDJBSmXtKq108Yhvbdh8NaMAzb0RXvtHDq68qibYRutrQ8DrVD7ekuX7r33dtjuaN_RlRtxgx0uykty1moXcPa3U_K2fngtHqPyefNULMtIcxEnEWY6wdzUnGGTNZDkaY46xgxa0RoGNdPGpJDpuklNJmNIRCPyWoo0RQaIOZ-S22N37_uvEcOgdjbU6JzusB-D4iA4z-PkV7w-ihYR1d7bnfbfCoSQMst-6c2R6nqnTN9_BgVMHe5TVaEO96lD4u5fSRlvseU_stZvYg |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK RIE RIL 7SC 8FD JQ2 L7M L~C L~D |
| DOI | 10.1109/SC.2005.4 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP All) 1998-Present Computer and Information Systems Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
| DatabaseTitle | Computer and Information Systems Abstracts Technology Research Database Computer and Information Systems Abstracts – Academic Advanced Technologies Database with Aerospace ProQuest Computer Science Collection Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Computer and Information Systems Abstracts |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science Government |
| EndPage | 25 |
| ExternalDocumentID | 1559977 |
| Genre | orig-research Conference Paper |
| GroupedDBID | 6IE 6IK 6IL AAJGR AARBI ACM ADPZR ALMA_UNASSIGNED_HOLDINGS APO BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK GUFHI IERZE OCL RIB RIC RIE RIL AAWTH LHSKQ 7SC 8FD JQ2 L7M L~C L~D |
| ID | FETCH-LOGICAL-a3524-e7a4e8bc30ed7d14868ea2e71f5fb01c0abb617acd6b792145d58c9566e01ee83 |
| IEDL.DBID | RIE |
| ISBN | 1595930612 9781595930613 |
| IngestDate | Fri Jul 11 01:05:32 EDT 2025 Wed Aug 27 01:43:52 EDT 2025 Wed Jan 31 06:49:19 EST 2024 Wed Jan 31 06:51:04 EST 2024 |
| IsDoiOpenAccess | false |
| IsOpenAccess | true |
| IsPeerReviewed | false |
| IsScholarly | false |
| Language | English |
| LinkModel | DirectLink |
| MeetingName | SC '05: International Conference for High Performance Computing, Networking, Storage and Analysis |
| MergedId | FETCHMERGED-LOGICAL-a3524-e7a4e8bc30ed7d14868ea2e71f5fb01c0abb617acd6b792145d58c9566e01ee83 |
| Notes | SourceType-Conference Papers & Proceedings-1 ObjectType-Conference Paper-1 content type line 25 |
| OpenAccessLink | https://www.osti.gov/biblio/919215 |
| PQID | 31533824 |
| PQPubID | 23500 |
| PageCount | 1 |
| ParticipantIDs | acm_books_10_1109_SC_2005_4 proquest_miscellaneous_31533824 acm_books_10_1109_SC_2005_4_brief ieee_primary_1559977 |
| PublicationCentury | 2000 |
| PublicationDate | 2005-00-00 |
| PublicationDateYYYYMMDD | 2005-01-01 |
| PublicationDate_xml | – year: 2005 text: 2005-00-00 |
| PublicationDecade | 2000 |
| PublicationPlace | Washington, DC, USA |
| PublicationPlace_xml | – name: Washington, DC, USA |
| PublicationSeriesTitle | ACM Conferences |
| PublicationTitle | Proceedings of the 2005 ACM/IEEE conference on Supercomputing |
| PublicationTitleAbbrev | SUPERC |
| PublicationYear | 2005 |
| Publisher | IEEE Computer Society IEEE |
| Publisher_xml | – name: IEEE Computer Society – name: IEEE |
| SSID | ssj0001969154 |
| Score | 1.6576896 |
| Snippet | Many emerging large-scale data science applications require searching large graphs distributed across multiple memories and processors. This paper presents a... |
| SourceID | proquest ieee acm |
| SourceType | Aggregation Database Publisher |
| StartPage | 25 |
| SubjectTerms | Area measurement Computer systems organization -- Architectures -- Distributed architectures -- Grid computing Computer systems organization -- Architectures -- Parallel architectures -- Multicore architectures Computer systems organization -- Architectures -- Serial architectures -- Superscalar architectures Computing methodologies -- Artificial intelligence -- Search methodologies Computing methodologies -- Artificial intelligence -- Search methodologies -- Discrete space search Computing methodologies -- Artificial intelligence -- Search methodologies -- Game tree search Computing methodologies -- Modeling and simulation -- Simulation types and techniques -- Massively parallel and high-performance simulations Extraterrestrial measurements Gain measurement Government Laboratories Large-scale systems Partitioning algorithms Research and development Scalability Software and its engineering -- Software organization and properties -- Software system structures -- Distributed systems organizing principles -- Grid computing Testing Theory of computation -- Design and analysis of algorithms -- Data structures design and analysis -- Sorting and searching |
| Title | A Scalable Distributed Parallel Breadth-First Search Algorithm on BlueGene/L |
| URI | https://ieeexplore.ieee.org/document/1559977 https://www.proquest.com/docview/31533824 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3fa9swED7asIc8tesPmnZbNdhj3diybMmPTdawh1IC3SBvQpIvbSC1ixPv759OcZqHjkLfbITAnCXdd3e67wP4obhJMStVlFkuIpEoE6nMFBG3zuUoLcfUBrEJeX-vZrNiugdXr70wiBgun-E1PYZaflm7llJlQyqhebyyD_tS5pterV0-pcgLDweodysjsl1y3R2l0_Y97ZiFkrgYPow3-RRBXsk9d9oqbw7k4GUmBx_7vkM42bXrsemrI_oMe1gdwcFWr4F12_cI-jth3WO4u_EDZkmdU-wnseeS8BWWbGoakldZspGHk-X6KZosPEBkm2vJ7Gb5WDeL9dMzqys2WrZItNXDuxP4M7n9Pf4VddIKkfGIS0QojUBlXRpjKUsfEuUKDUeZzLO5jRMXG2s9tjGuzK0siM28zJTzsVSOcYKo0lPoVXWFZ8C45XwuuA8TJRcSlcoLldlA86NsrtwALryBNcUMKx1CjrjQD2PSv8y0GMD3d0a1bRY4H8Ax2Vq_bPg3dGfmAVxuf5b2e4IKHabCul3plECs4uL8_xMvoB8IWEMi5Qv01k2LX-GT-7terJpvYVn9A2gkxjU |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1Ra9swED66brA-dWs7lnVrNdjjvNiyZEuPbdrQsSwE2kHfhCRf1kBqDyfp769OcZqHjUHfbIRBnCzdd3e67wP4orjNUVYqkY6LRGTKJkpanXDnfYGl45i7KDZRjsfq9lZPduDrUy8MIsbLZ_iNHmMtv2r8ilJlfSqhBbzyAl5KIXi67tbaZlR0oQMgoO4tSXS75Lw7UqfNe95xC2Wp7l8P1hkVQX7J33fqKn8dydHPDPefN8M3cLRt2GOTJ1f0FnawPoD9jWID6zbwAextpXUPYXQWBuyceqfYBfHnkvQVVmxiWxJYmbPzACir5V0ynAWIyNYXk9nZ_HfTzpZ396yp2fl8hURc3R8dwa_h5c3gKunEFRIbMJdIsLQClfN5ilVZhaCoUGg5ltlUTl2a-dQ6F9CN9VXhSk185pVUPkRTBaYZosrfwW7d1PgeGHecT8N6aFFyUaJShVbSRaIf5Qrle3AcDGwoaliYGHSk2lwPSAFTGtGDz_8ZNa6d4bQHh2Rr82fNwGE6M_fgdLNYJuwKKnXYGpvVwuQEYxUXH_794Sm8vrr5OTKj7-Mfx7AX6VhjWuUj7C7bFX6CV_5hOVu0J_EXewTntcl8 |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=SC+%2705+%3A+gateway+to+discovery+%3A+Washington+State+Convention+and+Trade+Center%2C+Seattle%2C+WA%2C+November+12-18%2C+2005&rft.atitle=A+Scalable+Distributed+Parallel+Breadth-First+Search+Algorithm+on+BlueGene%2FL&rft.au=Yoo%2C+A.&rft.au=Chow%2C+E.&rft.au=Henderson%2C+K.&rft.au=McLendon%2C+W.&rft.date=2005-01-01&rft.pub=IEEE&rft.isbn=9781595930613&rft.spage=25&rft.epage=25&rft_id=info:doi/10.1109%2FSC.2005.4&rft.externalDocID=1559977 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=9781595930613/lc.gif&client=summon&freeimage=true |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=9781595930613/mc.gif&client=summon&freeimage=true |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=9781595930613/sc.gif&client=summon&freeimage=true |