NEO-DNND: Communication-Optimized Distributed Nearest Neighbor Graph Construction

Graph-based approximate nearest neighbor algorithms have shown high neighbor structure representation quality. NN-Descent is a widely known graph-based approximate nearest neighbor (ANN) algorithm. However, graph-based approaches are memory- and time-consuming.To address the drawbacks, we develop a...

Full description

Saved in:
Bibliographic Details
Published in:SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis pp. 688 - 696
Main Authors: Iwabuchi, Keita, Steil, Trevor, Priest, Benjamin W., Pearce, Roger, Sanders, Geoffrey
Format: Conference Proceeding
Language:English
Published: IEEE 17.11.2024
Subjects:
approximate nearest neighbor
Approximation algorithms
Artificial neural networks
Conferences
distributed computing
Focusing
High performance computing
Libraries
Optimization
Random access memory
Scalability
Vectors
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Graph-based approximate nearest neighbor algorithms have shown high neighbor structure representation quality. NN-Descent is a widely known graph-based approximate nearest neighbor (ANN) algorithm. However, graph-based approaches are memory- and time-consuming.To address the drawbacks, we develop a scalable distributed NN-Descent. Our NEO-DNND (neighbor-checking efficiency optimized distributed NN-Descent) is built on top of MPI and designed to utilize network bandwidth efficiently. NEO-DNND reduces duplicate elements, increases intra-node data sharing, and leverages available DRAM to replicate data that may be sent frequently.NEO-DNND showed remarkable scalability up to 256 nodes and was able to construct neighborhood graphs from billion-scale datasets. Compared to a leading shared-memory ANN library, NEO-DNND achieved competitive performance even on a single node and exhibited 41.7X better performance by scaling up to 32 nodes. Furthermore, NEO-DNND outperformed a state-of-the-art distributed NN-Descent implementation, achieving up to a 6.0X speedup.
AbstractList Graph-based approximate nearest neighbor algorithms have shown high neighbor structure representation quality. NN-Descent is a widely known graph-based approximate nearest neighbor (ANN) algorithm. However, graph-based approaches are memory- and time-consuming.To address the drawbacks, we develop a scalable distributed NN-Descent. Our NEO-DNND (neighbor-checking efficiency optimized distributed NN-Descent) is built on top of MPI and designed to utilize network bandwidth efficiently. NEO-DNND reduces duplicate elements, increases intra-node data sharing, and leverages available DRAM to replicate data that may be sent frequently.NEO-DNND showed remarkable scalability up to 256 nodes and was able to construct neighborhood graphs from billion-scale datasets. Compared to a leading shared-memory ANN library, NEO-DNND achieved competitive performance even on a single node and exhibited 41.7X better performance by scaling up to 32 nodes. Furthermore, NEO-DNND outperformed a state-of-the-art distributed NN-Descent implementation, achieving up to a 6.0X speedup.
Author Iwabuchi, Keita
Sanders, Geoffrey
Pearce, Roger
Steil, Trevor
Priest, Benjamin W.
Author_xml – sequence: 1
  givenname: Keita
  surname: Iwabuchi
  fullname: Iwabuchi, Keita
  email: kiwabuchi@llnl.gov
  organization: Lawrence Livermore National Laboratory,Center for Applied Scientific Computing
– sequence: 2
  givenname: Trevor
  surname: Steil
  fullname: Steil, Trevor
  email: steil1@llnl.gov
  organization: Lawrence Livermore National Laboratory,Center for Applied Scientific Computing
– sequence: 3
  givenname: Benjamin W.
  surname: Priest
  fullname: Priest, Benjamin W.
  email: priest2@llnl.gov
  organization: Lawrence Livermore National Laboratory,Center for Applied Scientific Computing
– sequence: 4
  givenname: Roger
  surname: Pearce
  fullname: Pearce, Roger
  email: rpearce@llnl.gov
  organization: Lawrence Livermore National Laboratory,Center for Applied Scientific Computing
– sequence: 5
  givenname: Geoffrey
  surname: Sanders
  fullname: Sanders, Geoffrey
  email: sanders29@llnl.gov
  organization: Lawrence Livermore National Laboratory,Center for Applied Scientific Computing
BookMark eNotj8tKxEAURFtQUMd8gS7yA4m3n-l2J8k4CkOCqLgcOp0bp8E86CQL_XojuqpanFNQl-S0H3ok5JpCSimY25f8XXEmIGXARAoARp2QyGRGcwlcSin4OYmmydegQGoBWl6Q53JbJUVZFndxPnTd0ntnZz_0STXOvvPf2MSFn-bg62Vee4k24DSv6T-O9RDiXbDjcVX7lVncr3lFzlr7OWH0nxvy9rB9zR-TfbV7yu_3iWVSzYkRGdZUIwoQSjuL3GmKDA3PHG8kR2VZy1kjVcZbqgDq1jTO1UpCpnA9tCE3f7seEQ9j8J0NXwcKmq0A5z_TLFCm
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/SCW63240.2024.00096
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Xplore POP ALL
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
EISBN 9798350355543
EndPage 696
ExternalDocumentID 10820763
Genre orig-research
GrantInformation_xml – fundername: U.S. Department of Energy
  funderid: 10.13039/100000015
GroupedDBID 6IE
6IL
ACM
ALMA_UNASSIGNED_HOLDINGS
CBEJK
RIE
RIL
ID FETCH-LOGICAL-a256t-947eb18ee40468cae3c81e2e937c3d53e6a2f32d5673f1600bf9dccb65076e503
IEDL.DBID RIE
ISICitedReferencesCount 0
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001451792300074&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 01:59:34 EDT 2025
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a256t-947eb18ee40468cae3c81e2e937c3d53e6a2f32d5673f1600bf9dccb65076e503
PageCount 9
ParticipantIDs ieee_primary_10820763
PublicationCentury 2000
PublicationDate 2024-Nov.-17
PublicationDateYYYYMMDD 2024-11-17
PublicationDate_xml – month: 11
  year: 2024
  text: 2024-Nov.-17
  day: 17
PublicationDecade 2020
PublicationTitle SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis
PublicationTitleAbbrev SC-W
PublicationYear 2024
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssib060584085
Score 1.8897113
Snippet Graph-based approximate nearest neighbor algorithms have shown high neighbor structure representation quality. NN-Descent is a widely known graph-based...
SourceID ieee
SourceType Publisher
StartPage 688
SubjectTerms approximate nearest neighbor
Approximation algorithms
Artificial neural networks
Conferences
distributed computing
Focusing
High performance computing
Libraries
Optimization
Random access memory
Scalability
Vectors
Title NEO-DNND: Communication-Optimized Distributed Nearest Neighbor Graph Construction
URI https://ieeexplore.ieee.org/document/10820763
WOSCitedRecordID wos001451792300074&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlZ09T8MwEIYtihiYAFHEtzKwGmzHiR3WfsCUFgGiW-XYZ6kDLaKFgV_PndtSMTAwxUqGKOfY9zrO-xxjV6hSCy-k5yoKzTV5e6pYKB4k8ca8s4VrUrEJU9d2NKqGK7N68sIAQPr5DK6pmfbyw8x_0KcyHOGYr3BAtFjLGLM0a61fHtreI1rXiiwkRXXz2HkhGLnAVaAiRrZIZP5NDZWUQvp7_7z5PmtvzHjZ8CfNHLAtmB6yh7o34N267t5mvywefIBTwOvkC0LWJSQuVbPCdk2k2vkCj7gWx17P7ohTnVG1zjU_ts2e-72nzj1fVUfgDmXKglfa4DxrATQuca13kHsrQQHqDZ-HIofSqZirUJQmjxJ1TROr4H2DksyUUIj8iG1PZ1M4ZlkQznkXTYnaQeuonTJBNNo2eAFktCesTfEYvy0BGON1KE7_OH_GdinkZNmT5pxt45PABdvxn4vJ_P0ydds39cKZBw
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlZ27T8MwEMYtKEgwAaKINxlYDbbjvFj7oIiSFlFEt8qxz1IHUkQLA389d2lLxcDAFCsZopxj3-c43-8Yu0SVGlkhLVdeaK7J25P5SHEniTdmTRqZoio2keR5Ohxm_YVZvfLCAED18xlcUbPay3cT-0GfynCEY77CAbHONiKtlZzbtZavD23wEa9rwRaSIrt-arwQjlzgOlARJVtUbP5VFZUqibR3_nn7XVZf2fGC_k-i2WNrUO6zx7zV4808b94Ev0wevIeTwOv4C1zQJCgu1bPCdk6s2ukMj7gax34PbolUHVC9ziVBts6e261Bo8MX9RG4QaEy45lOcKZNATQuclNrILSpBAWoOGzoohBio3yoXBQnoZeobAqfOWsLFGVJDJEID1itnJRwyAInjLHGJzGqB629NipxotBpgRdA-vSI1Skeo7c5AmO0DMXxH-cv2FZn8NAdde_y-xO2TeEnA59MTlkNnwrO2Kb9nI2n7-dVF34DGzicTg
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=SC24-W%3A+Workshops+of+the+International+Conference+for+High+Performance+Computing%2C+Networking%2C+Storage+and+Analysis&rft.atitle=NEO-DNND%3A+Communication-Optimized+Distributed+Nearest+Neighbor+Graph+Construction&rft.au=Iwabuchi%2C+Keita&rft.au=Steil%2C+Trevor&rft.au=Priest%2C+Benjamin+W.&rft.au=Pearce%2C+Roger&rft.date=2024-11-17&rft.pub=IEEE&rft.spage=688&rft.epage=696&rft_id=info:doi/10.1109%2FSCW63240.2024.00096&rft.externalDocID=10820763