Unified Communication Optimization Strategies for Sparse Triangular Solver on CPU and GPU Clusters

This paper presents a unified communication optimization frame-work for sparse triangular solve (SpTRSV) algorithms on CPU and GPU clusters. The framework builds upon a 3D communication-avoiding (CA) layout of P_{x}\times P_{y}\times P_{z} processes that divides a sparse matrix into P_{z} submatrice...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:International Conference for High Performance Computing, Networking, Storage and Analysis (Online) S. 1 - 15
Hauptverfasser: Liu, Yang, Ding, Nan, Sao, Piyush, Williams, Samuel, Li, Xiaoye Sherry
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: ACM 11.11.2023
Schlagworte:
Clustering algorithms
communication optimization
communication-avoiding algorithm
Graphics processing units
High performance computing
Layout
NVSH-MEM
Parallel processing
Scalability
sparse matrix
SpTRSV
supernodal method
Three-dimensional displays
triangular solve
ISSN:2167-4337
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:This paper presents a unified communication optimization frame-work for sparse triangular solve (SpTRSV) algorithms on CPU and GPU clusters. The framework builds upon a 3D communication-avoiding (CA) layout of P_{x}\times P_{y}\times P_{z} processes that divides a sparse matrix into P_{z} submatrices, each handled by a P_{x}\times P_{y}2\mathrm{D} grid with block-cyclic distribution. We propose three communication optimization strategies: First, a new 3D SpTRSV algorithm is developed, which trades the inter-grid communication and synchronization with replicated computation. This design requires only one inter-grid synchronization, and the inter-grid communication is efficiently implemented with sparse allreduce operations. Second, broadcast and reduction communication trees are used to reduce message latency of the intra-grid 2D communication on CPU clus-ters. Finally, we leverage GPU-initiated one-sided communication to implement the communication trees on GPU clusters. With these nested inter- and intra-grid communication optimization strategies, the proposed 3D SpTRSV algorithm can attain up to 3.45x speedups compared to the baseline 3D SpTRSV algorithm using up to 2048 Cori Haswell CPU cores. In addition, the proposed GPU 3D Sp-TRSV algorithm can achieve up to 6.5x speedups compared to the proposed CPU 3D SpTRSV algorithm with P_{z} up to 64. Finally it is remarkable that the proposed GPU 3D SpTRSV can scale to 256 GPUs using the Perlmutter system while the existing 2D SpTRSV algorithm can only scale up to 4 GPUs.
ISSN:2167-4337
DOI:10.1145/3581784.3607092