UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programming

Distributed learning is commonly used for training deep learning models, especially large models. In distributed learning, manual parallelism (MP) methods demand considerable human effort and have limited flexibility. Hence, automatic parallelism (AP) methods have recently been proposed for automati...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 20947 - 20957
Main Authors: Lin, Hao, Wu, Ke, Li, Jie, Li, Jun, Li, Wu-Jun
Format: Conference Proceeding
Language:English
Published: IEEE 10.06.2025
Subjects:
Computer aided instruction
Distance learning
Manuals
Optimization
Parallel processing
Pattern recognition
Quadratic programming
Throughput
Training
Transformers
ISSN:1063-6919
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Distributed learning is commonly used for training deep learning models, especially large models. In distributed learning, manual parallelism (MP) methods demand considerable human effort and have limited flexibility. Hence, automatic parallelism (AP) methods have recently been proposed for automating the parallel strategy optimization process. Existing AP methods suffer from sub-optimal solutions because they do not jointly optimize the two categories of parallel strategies (i.e., inter-layer parallelism and intra-layer parallelism). In this paper, we propose a novel AP method called UniAP, which unifies inter- and intra-layer automatic parallelism by mixed integer quadratic programming. To the best of our knowledge, UniAP is the first parallel method that can jointly optimize the two categories of parallel strategies to find an optimal solution. Experimental results show that UniAP outperforms state-of-the-art methods by up to 3.80× in throughput and reduces strategy optimization time by up to 107× across five Transformer-Based models.
AbstractList Distributed learning is commonly used for training deep learning models, especially large models. In distributed learning, manual parallelism (MP) methods demand considerable human effort and have limited flexibility. Hence, automatic parallelism (AP) methods have recently been proposed for automating the parallel strategy optimization process. Existing AP methods suffer from sub-optimal solutions because they do not jointly optimize the two categories of parallel strategies (i.e., inter-layer parallelism and intra-layer parallelism). In this paper, we propose a novel AP method called UniAP, which unifies inter- and intra-layer automatic parallelism by mixed integer quadratic programming. To the best of our knowledge, UniAP is the first parallel method that can jointly optimize the two categories of parallel strategies to find an optimal solution. Experimental results show that UniAP outperforms state-of-the-art methods by up to 3.80× in throughput and reduces strategy optimization time by up to 107× across five Transformer-Based models.
Author Li, Jun
Lin, Hao
Li, Wu-Jun
Wu, Ke
Li, Jie
Author_xml – sequence: 1
  givenname: Hao
  surname: Lin
  fullname: Lin, Hao
  email: hao.lin@smail.nju.edu.cn
  organization: Nanjing University,National Key Laboratory for Novel Software Technology, School of Computer Science,Nanjing,China,210023
– sequence: 2
  givenname: Ke
  surname: Wu
  fullname: Wu, Ke
  email: ke.wu@smail.nju.edu.cn
  organization: Nanjing University,National Key Laboratory for Novel Software Technology, School of Computer Science,Nanjing,China,210023
– sequence: 3
  givenname: Jie
  surname: Li
  fullname: Li, Jie
  email: jie-li@smail.nju.edu.cn
  organization: Nanjing University,National Key Laboratory for Novel Software Technology, School of Computer Science,Nanjing,China,210023
– sequence: 4
  givenname: Jun
  surname: Li
  fullname: Li, Jun
  email: lijun@smail.nju.edu.cn
  organization: Nanjing University,National Key Laboratory for Novel Software Technology, School of Computer Science,Nanjing,China,210023
– sequence: 5
  givenname: Wu-Jun
  surname: Li
  fullname: Li, Wu-Jun
  email: liwujun@nju.edu.cn
  organization: Nanjing University,National Key Laboratory for Novel Software Technology, School of Computer Science,Nanjing,China,210023
BookMark eNotkMtOwzAQRQ0CiVL6B134B1JsT-LY7KoKSqUiwnNbTexJZZSkyEkl8ve0lNW5i6OzuNfsot21xNhUipmUwt4uPovXTOWQzpRQ2UxIm8kzNrG5NQAyS0Gn5pyNpNCQaCvtFZt03ZcQApSU2poRcx9tmBd3_IBqCO2Wr9qeYsKx9ccZMVnjQJHP9_2uwT44XmDEuqY6dA0vB_4UfuhPpe1Be9mjjyct7rYRm-bQvGGXFdYdTf45Zm8P9--Lx2T9vFwt5uskWOgTrZEMegteSNTCQ14CGg8GVI5ee_IuQ3Jk0aE0RqS60lQqKFXq8srBmE1P1UBEm-8YGozD5ngTgJbwC4iJWYk
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/CVPR52734.2025.01951
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Applied Sciences
EISBN 9798331543648
EISSN 1063-6919
EndPage 20957
ExternalDocumentID 11093361
Genre orig-research
GroupedDBID 6IE
6IH
6IL
6IN
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IJVOP
OCL
RIE
RIL
RIO
ID FETCH-LOGICAL-i93t-66ae8ad93d01a60d37b3a8d38327ad6dedc5aece9aca188046f6eb23b24c7fc3
IEDL.DBID RIE
IngestDate Wed Aug 20 06:20:57 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i93t-66ae8ad93d01a60d37b3a8d38327ad6dedc5aece9aca188046f6eb23b24c7fc3
PageCount 11
ParticipantIDs ieee_primary_11093361
PublicationCentury 2000
PublicationDate 2025-June-10
PublicationDateYYYYMMDD 2025-06-10
PublicationDate_xml – month: 06
  year: 2025
  text: 2025-June-10
  day: 10
PublicationDecade 2020
PublicationTitle Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online)
PublicationTitleAbbrev CVPR
PublicationYear 2025
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0003211698
Score 2.2941923
Snippet Distributed learning is commonly used for training deep learning models, especially large models. In distributed learning, manual parallelism (MP) methods...
SourceID ieee
SourceType Publisher
StartPage 20947
SubjectTerms Computer aided instruction
Distance learning
Manuals
Optimization
Parallel processing
Pattern recognition
Quadratic programming
Throughput
Training
Transformers
Title UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programming
URI https://ieeexplore.ieee.org/document/11093361
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwELagYmAqjyLe8sAamsSJH2xVRcUAVXgIdavOj6BINEVpguDfYzuhsDAwxYpOtuST5fvO932H0IWSPInjnAVAcwtQkggCoR1XJqGp5hHhEEnfbIJNp3w2E1lHVvdcGGOMLz4zl27o3_L1UjUuVTZ06piEOLCzyRhtyVrrhAqxUIYK3tHjrOVw_Jw9OH0xlzqJnTqncM-Rv5qo-Dtk0v_n6jto8MPGw9n6ntlFG6bcQ_0ufMTd4VztI2Xjx1F2he3Hk5ewz_YFGErthhUEt2ADbDxq6qUXasUZVK6VymuxWmD5ie-KD-NNzYs1u29AV61ZW8O1sHMO0OPk-ml8E3Q9FIJCkDqgFAwHLYgOI6ChJkwS4NrC0piBptpolYJRRoACp8yW0JxarE1knCiWK3KAeuWyNIcI85RKkSie61QkCTMSrF9zFkYqT2UcwxEauC2bv7UiGfPv3Tr-4_8J2nZecVVXUXiKenXVmDO0pd7rYlWde9d-Aemvphw
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3NS8MwFA8yBT3Nj4nf5uC1bm3aNPE2hmPiNqoO2W28fFQKrpOuFf3vTbI6vXjw1FAeCeQR8t7L-_1-CF1JwcIgSGMPaGoSlNAHjyuLlQlppJhPGPjCiU3E4zGbTnlSg9UdFkZr7ZrP9LUdurd8tZCVLZW1LTsmITbZ2bTSWTVca11SISaZoZzVADlj2-49J4-WYcwWTwLLz8ntg-QvGRV3i_Sb_1x_F7V-8Hg4Wd80e2hD5_uoWQeQuD6eywMkTQTZTW6w-Tj4Enb1Pg9DruywAG8IJsTG3apcOKpWnEBhxVRes-Uci088yj60M9UvxuyhAlWszFZdXHMzZws99W8nvYFXqyh4GSelRyloBooT1fGBdhSJBQGmTGIaxKCo0kpGoKXmIMFys4U0pSbbJiIIZZxKcoga-SLXRwiziAoeSpaqiIdhrAUYz6Zxx5dpJIIAjlHLbtnsbUWTMfverZM__l-i7cFkNJwN78b3p2jHesj2YPmdM9Qoi0qfoy35XmbL4sK5-Qu9lKll
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+%28IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition.+Online%29&rft.atitle=UniAP%3A+Unifying+Inter-+and+Intra-Layer+Automatic+Parallelism+by+Mixed+Integer+Quadratic+Programming&rft.au=Lin%2C+Hao&rft.au=Wu%2C+Ke&rft.au=Li%2C+Jie&rft.au=Li%2C+Jun&rft.date=2025-06-10&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=20947&rft.epage=20957&rft_id=info:doi/10.1109%2FCVPR52734.2025.01951&rft.externalDocID=11093361