Embracing Irregular Parallelism in HPC with YGM

YGM is a general-purpose asynchronous distributed computing library for C++/MPI, designed to handle the irregular data access patterns and small messages of graph algorithms and data science applications. It uses data serialization to give an easily usable active message interface and message aggreg...

Full description

Saved in:
Bibliographic Details
Published in:International Conference for High Performance Computing, Networking, Storage and Analysis (Online) pp. 1 - 13
Main Authors: Steil, Trevor, Reza, Tahsin, Priest, Benjamin W., Pearce, Roger
Format: Conference Proceeding
Language:English
Published: ACM 11.11.2023
Subjects:
asynchronous communication
Benchmark testing
Costs
distributed computing
Programming
Random access memory
Scalability
Social networking (online)
Throughput
ISSN:2167-4337
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract YGM is a general-purpose asynchronous distributed computing library for C++/MPI, designed to handle the irregular data access patterns and small messages of graph algorithms and data science applications. It uses data serialization to give an easily usable active message interface and message aggregation to maximize application throughput. Our design philosophy makes a tradeoff that increases network bandwidth utilization at the cost of added latency. We provide a suite of benchmarks showcasing YGM's performance. Compared to similar distributed active message benchmark implementations that do not provide message buffering, we are able to achieve over 10x throughput on thousands of cores at a latency cost that can be as small as 2x or as large as 100x, depending on the machine being used. For applications that can be written to be latency-tolerant, this represents a significant potential performance improvement through using YGM.
AbstractList YGM is a general-purpose asynchronous distributed computing library for C++/MPI, designed to handle the irregular data access patterns and small messages of graph algorithms and data science applications. It uses data serialization to give an easily usable active message interface and message aggregation to maximize application throughput. Our design philosophy makes a tradeoff that increases network bandwidth utilization at the cost of added latency. We provide a suite of benchmarks showcasing YGM's performance. Compared to similar distributed active message benchmark implementations that do not provide message buffering, we are able to achieve over 10x throughput on thousands of cores at a latency cost that can be as small as 2x or as large as 100x, depending on the machine being used. For applications that can be written to be latency-tolerant, this represents a significant potential performance improvement through using YGM.
Author Reza, Tahsin
Pearce, Roger
Steil, Trevor
Priest, Benjamin W.
Author_xml – sequence: 1
  givenname: Trevor
  surname: Steil
  fullname: Steil, Trevor
  email: steil1@llnl.gov
  organization: Center for Applied Scientific Computing (CASC),Lawrence Livermore National Laboratory (LLNL),Livermore,CA,USA
– sequence: 2
  givenname: Tahsin
  surname: Reza
  fullname: Reza, Tahsin
  email: tahsin.reza@uwaterloo.ca
  organization: University of Waterloo,Department of Electrical and Computer Engineering,Waterloo,Ontario,Canada
– sequence: 3
  givenname: Benjamin W.
  surname: Priest
  fullname: Priest, Benjamin W.
  email: priest2@llnl.gov
  organization: Center for Applied Scientific Computing (CASC),Lawrence Livermore National Laboratory (LLNL),Livermore,CA,USA
– sequence: 4
  givenname: Roger
  surname: Pearce
  fullname: Pearce, Roger
  email: rpearce@llnl.gov
  organization: Center for Applied Scientific Computing (CASC),Lawrence Livermore National Laboratory (LLNL),Livermore,CA,USA
BookMark eNotjD1PwzAUAA0qEqVkZmHwH0j7XvzijxFFpa1URIcyMFUviV2MkoCcIsS_pxLohtMtdyMmw8fghbhDmCNSuVClRWNprjQYBHUhMmecJQADCK64FNMCtclJKXMtsnF8BwBVAKGFqVgs-zpxE4ej3KTkj18dJ7njxF3nuzj2Mg5yvavkdzy9ydfV0624CtyNPvv3TLw8LvfVOt8-rzbVwzZnZc0pN4QlOF2jYmggNAGtBschtN4VLXmrtGuZnaO6DKam2rdETXHmXIFaNRP3f9_ovT98pthz-jkgkC3BavULvL9E3w
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1145/3581784.3607103
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Xplore POP ALL
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISBN 9798400701092
EISSN 2167-4337
EndPage 13
ExternalDocumentID 10485086
Genre orig-research
GrantInformation_xml – fundername: U.S. Department of Energy
  grantid: DE-AC52-07NA27344,LLNL-CONF-853600
  funderid: 10.13039/100000015
GroupedDBID 6IE
6IF
6IH
6IK
6IL
6IN
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IPLJI
OCL
RIE
RIL
ID FETCH-LOGICAL-a387t-7415096b13a0c0fcf18609affde92d4e8369daa994b5f7b4bed44c2c2cf7bf4d3
IEDL.DBID RIE
ISICitedReferencesCount 3
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001461755900038&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:09:35 EDT 2025
IsDoiOpenAccess false
IsOpenAccess true
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a387t-7415096b13a0c0fcf18609affde92d4e8369daa994b5f7b4bed44c2c2cf7bf4d3
OpenAccessLink https://dl.acm.org/doi/pdf/10.1145/3581784.3607103
PageCount 13
ParticipantIDs ieee_primary_10485086
PublicationCentury 2000
PublicationDate 2023-Nov.-11
PublicationDateYYYYMMDD 2023-11-11
PublicationDate_xml – month: 11
  year: 2023
  text: 2023-Nov.-11
  day: 11
PublicationDecade 2020
PublicationTitle International Conference for High Performance Computing, Networking, Storage and Analysis (Online)
PublicationTitleAbbrev SC
PublicationYear 2023
Publisher ACM
Publisher_xml – name: ACM
SSID ssj0003204180
ssib053141430
Score 1.8941408
Snippet YGM is a general-purpose asynchronous distributed computing library for C++/MPI, designed to handle the irregular data access patterns and small messages of...
SourceID ieee
SourceType Publisher
StartPage 1
SubjectTerms asynchronous communication
Benchmark testing
Costs
distributed computing
Programming
Random access memory
Scalability
Social networking (online)
Throughput
Title Embracing Irregular Parallelism in HPC with YGM
URI https://ieeexplore.ieee.org/document/10485086
WOSCitedRecordID wos001461755900038&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV27TsMwFLVoxcBUHkVQHvLAmjaurxN7rlqKBFUGQGWq_LiRKtEUpS3fj50mwMKAvNiWB799r-1zDiF3KjWKobORNwVEBEzbSBprIy01GKNyriv42OtjOpvJ-VxlNVi9wsIgYvX5DPshWr3lu7Xdhasyv8JBeoMiaZFWmiZ7sFYzefxcAtZQiYdtmA9jYDKu6XwYiEGg-kol9HngVAsqWb_0VKrjZNL5Z0WOSfcHmEez7yPnhBxgcUo6jTIDrRfqGRmMV94Ltr4MfSjLSm6-pJkug3DK-3KzosuCTrMRDbew9O3-qUteJuPn0TSqpREizWW6jYId4J0Pw7iObZzbnMkkVjrPHaqhA5Q8UU5rpcCIPDVg0AHYoQ8-lYPj56RdrAu8IJTbGB0aLZgFECik0sJywa0TiXdv8JJ0QwcsPvbsF4um7b0_8q_IUZBkD3g9xq5Je1vu8IYc2s_tclPeVmP2Bbqwli4
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NT8IwGG4UTfSEHxi_7cHrYF3fbu2ZgBCB7IAGT6RfS0hkmAH-ftuxqRcPppe26aHffd-2z_Mg9CgSJYg1OnCmAAuASB1wpXUguQSlREZlCR97HSWTCZ_NRFqB1UssjLW2_Hxm2z5avuWbld76qzK3woE7gyLeRwcMIAp3cK16-rjZBKQmE_cbMY1CIDysCH0IsI4n-0o4tKlnVfM6Wb8UVcoDpd_8Z1VOUOsHmofT70PnFO3Z_Aw1a20GXC3Vc9TpLZ0frF0ZPCyKUnC-wKksvHTK-2K9xIscD9Iu9vew-O1p3EIv_d60OwgqcYRAUp5sAm8JOPdDESpDHWY6IzwOhcwyY0VkwHIaCyOlEKBYlihQ1gDoyAWXysDQC9TIV7m9RJjq0BqrJCMagFnGhWSaMqoNi52DY69Qy3fA_GPHfzGv2379R_4DOhpMx6P5aDh5vkHHXqDdo_cIuUWNTbG1d-hQf24W6-K-HL8vv5KZdQ
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=International+Conference+for+High+Performance+Computing%2C+Networking%2C+Storage+and+Analysis+%28Online%29&rft.atitle=Embracing+Irregular+Parallelism+in+HPC+with+YGM&rft.au=Steil%2C+Trevor&rft.au=Reza%2C+Tahsin&rft.au=Priest%2C+Benjamin+W.&rft.au=Pearce%2C+Roger&rft.date=2023-11-11&rft.pub=ACM&rft.eissn=2167-4337&rft.spage=1&rft.epage=13&rft_id=info:doi/10.1145%2F3581784.3607103&rft.externalDocID=10485086