SMSG: Profiling-Free Parallelism Modeling for Distributed Training of DNN

The increasing size of deep neural networks (DNNs) raises a high demand for distributed training. An expert could find good hybrid parallelism strategies, but designing suitable strategies is time and labor-consuming. Therefore, automating parallelism strategy generation is crucial and desirable for...

Celý popis

Uloženo v:

Podrobná bibliografie
Vydáno v:	International journal of parallel programming Ročník 51; číslo 2-3; s. 109 - 127
Hlavní autoři:	Wang, Haoran, Tachon, Thibaut, Li, Chong, Robert, Sophie, Limet, Sébastien
Médium:	Journal Article
Jazyk:	angličtina
Vydáno:	New York Springer US 01.06.2023 Springer Nature B.V Springer Verlag
Témata:	Artificial neural networks Communication Computation Computer Science Cost analysis Cost function Homomorphisms Mathematical analysis Parallel processing Portability Processor Architectures Searching Semantics Software Engineering/Programming and Operating Systems Special Issue on High-Level Parallel Programming and Applications (HLPP 2022) Theory of Computation Training Parallel modeling Distributed training Deep network networks Performance analysis Functional transformation Symbolic cost model Distributed training ; Deep network networks; Parallel modeling; Symbolic cost model; Functional transformation; Performance analysis
ISSN:	0885-7458, 1573-7640
On-line přístup:	Získat plný text
Tagy:	Přidat tag Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!

Abstract	The increasing size of deep neural networks (DNNs) raises a high demand for distributed training. An expert could find good hybrid parallelism strategies, but designing suitable strategies is time and labor-consuming. Therefore, automating parallelism strategy generation is crucial and desirable for DNN designers. Some automatic searching approaches have recently been studied to free the experts from the heavy parallel strategy conception. However, these approaches all rely on a numerical cost model, which requires heavy profiling results that lack portability. These profiling-based approaches cannot lighten the strategy generation work due to the non-reusable profiling value. Our intuition is that there is no need to estimate the actual execution time of the distributed training but to compare the relative cost of different strategies. We propose SMSG (Symbolic Modeling for Strategy Generation), which analyses the cost based on the communication and computation semantics. With SMSG, the parallel cost analyses are decoupled from hardware characteristics. SMSG defines cost functions for each kind of operator to quantitatively evaluate the amount of data for computation and communication, which eliminates the heavy profiling tasks. Besides, SMSG introduces how to apply functional transformation by using the Third Homomorphism theorem to control the high searching complexity. Our experiments show that SMSG can find good hybrid parallelism strategies to generate an efficient training performance similar to the state of the art. Moreover, SMSG covers a wide variety of DNN models with good scalability. SMSG provides good portability when changing training configurations that a profiling-based approach cannot.
AbstractList	Abstract The increasing size of deep neural networks (DNNs) raises a high demand for distributed training. An expert could find good hybrid parallelism strategies, but designing suitable strategies is time and labor-consuming. Therefore, automating parallelism strategy generation is crucial and desirable for DNN designers. Some automatic searching approaches have recently been studied to free the experts from the heavy parallel strategy conception. However, these approaches all rely on a numerical cost model, which requires heavy profiling results that lack portability. These profiling-based approaches cannot lighten the strategy generation work due to the non-reusable profiling value. Our intuition is that there is no need to estimate the actual execution time of the distributed training but to compare the relative cost of different strategies. We propose SMSG (Symbolic Modeling for Strategy Generation), which analyses the cost based on the communication and computation semantics. With SMSG, the parallel cost analyses are decoupled from hardware characteristics. SMSG defines cost functions for each kind of operator to quantitatively evaluate the amount of data for computation and communication, which eliminates the heavy profiling tasks. Besides, SMSG introduces how to apply functional transformation by using the Third Homomorphism theorem to control the high searching complexity. Our experiments show that SMSG can find good hybrid parallelism strategies to generate an efficient training performance similar to the state of the art. Moreover, SMSG covers a wide variety of DNN models with good scalability. SMSG provides good portability when changing training configurations that a profiling-based approach cannot. The increasing size of deep neural networks (DNNs) raises a high demand for distributed training. An expert could find good hybrid parallelism strategies, but designing suitable strategies is time and labor-consuming. Therefore, automating parallelism strategy generation is crucial and desirable for DNN designers. Some automatic searching approaches have recently been studied to free the experts from the heavy parallel strategy conception. However, these approaches all rely on a numerical cost model, which requires heavy profiling results that lack portability. These profiling-based approaches cannot lighten the strategy generation work due to the non-reusable profiling value. Our intuition is that there is no need to estimate the actual execution time of the distributed training but to compare the relative cost of different strategies. We propose SMSG (Symbolic Modeling for Strategy Generation), which analyses the cost based on the communication and computation semantics. With SMSG, the parallel cost analyses are decoupled from hardware characteristics. SMSG defines cost functions for each kind of operator to quantitatively evaluate the amount of data for computation and communication, which eliminates the heavy profiling tasks. Besides, SMSG introduces how to apply functional transformation by using the Third Homomorphism theorem to control the high searching complexity. Our experiments show that SMSG can find good hybrid parallelism strategies to generate an efficient training performance similar to the state of the art. Moreover, SMSG covers a wide variety of DNN models with good scalability. SMSG provides good portability when changing training configurations that a profiling-based approach cannot. The increasing size of deep neural networks (DNNs) raises a high demand for distributed training. An expert could find good hybrid parallelism strategies, but designing suitable strategies is time and labor-consuming. Therefore, automating parallelism strategy generation is crucial and desirable for DNN designers. Some automatic searching approaches have recently been studied to free the experts from the heavy parallel strategy conception. However, these approaches all rely on a numerical cost model, which requires heavy profiling results that lack portability. These profiling-based approaches cannot lighten the strategy generation work due to the non-reusable profiling value. Our intuition is that there is no need to estimate the actual execution time of the distributed training but to compare the relative cost of different strategies. We propose SMSG (Symbolic Modeling for Strategy Generation), which analyses the cost based on the communication and computation semantics. With SMSG, the parallel cost analyses are decoupled from hardware characteristics. SMSG defines cost functions for each kind of operator to quantitatively evaluate the amount of data for computation and communication, which eliminates the heavy profiling tasks. Besides, SMSG introduces how to apply functional transformation by using the Third Homomorphism theorem to control the high searching complexity. Our experiments show that SMSG can find good hybrid parallelism strategies to generate an efficient training performance similar to the state of the art. Moreover, SMSG covers a wide variety of DNN models with good scalability. SMSG provides good portability when changing training configurations that a profiling-based approach cannot.
Author	Li, Chong Wang, Haoran Robert, Sophie Tachon, Thibaut Limet, Sébastien
Author_xml	– sequence: 1 givenname: Haoran surname: Wang fullname: Wang, Haoran organization: Huawei Technologies France S.A.S.U., LIFO, Bat. 3IA, Université d’Orléans – sequence: 2 givenname: Thibaut surname: Tachon fullname: Tachon, Thibaut organization: Huawei Technologies France S.A.S.U – sequence: 3 givenname: Chong surname: Li fullname: Li, Chong email: ch.l@huawei.com organization: Huawei Technologies France S.A.S.U – sequence: 4 givenname: Sophie surname: Robert fullname: Robert, Sophie organization: LIFO, Bat. 3IA, Université d’Orléans – sequence: 5 givenname: Sébastien surname: Limet fullname: Limet, Sébastien organization: LIFO, Bat. 3IA, Université d’Orléans
BackLink	https://univ-orleans.hal.science/hal-04551101$$DView record in HAL
BookMark	eNp9kE1LxDAURYMoOKP-AVcFVy6iL0nzUXfi-AUzKqjrkLaJRjqNJh3Bf2_GKoILVyF554T77hRt9qG3CO0TOCIA8jgRkEJgoBTna0mw2EATwiXDUpSwiSagFMey5GobTVN6AYBKKjVB1_eL-8uT4i4G5zvfP-GLaG1xZ6LpOtv5tCwWobXrSeFCLGY-DdHXq8G2xUM0vl8PgitmNze7aMuZLtm973MHPV6cP5xd4fnt5fXZ6Rw3rJIDrkhLGC9rUVfWypa6umIgpKOlEi0IxoSTvHFGGtYSUzFpbSuZBOIq47g1bAcdjv8-m06_Rr808UMH4_XV6Vyv36DknBAg7ySzByP7GsPbyqZBv4RV7HM8TWVFJVXAaabUSDUxpBSt040fzOBDP-QVO01Ar0vWY8k6l6y_StYiq_SP-pPoX4mNUspw_2Tjb6p_rE_MdI6n
CitedBy_id	crossref_primary_10_1109_TPDS_2024_3476390
Cites_doi	10.1017/S0956796897002864 10.1145/1594834.1480905 10.1016/j.jcss.2010.06.012 10.1109/TPDS.2021.3132413 10.1017/S0956796800001908 10.1007/978-3-030-85665-6_13 10.1145/3437801.3441593 10.1145/2988450.2988454 10.1109/CVPR.2016.90
ContentType	Journal Article
Copyright	The Author(s) 2022 The Author(s) 2022. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. Distributed under a Creative Commons Attribution 4.0 International License
Copyright_xml	– notice: The Author(s) 2022 – notice: The Author(s) 2022. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. – notice: Distributed under a Creative Commons Attribution 4.0 International License
DBID	C6C AAYXX CITATION 3V. 7SC 7WY 7WZ 7XB 87Z 8AL 8FD 8FE 8FG 8FK 8FL 8G5 ABUWG AFKRA ARAPS AZQEC BENPR BEZIV BGLVJ CCPQU DWQXO FRNLG F~G GNUQQ GUQSH HCIFZ JQ2 K60 K6~ K7- L.- L7M L~C L~D M0C M0N M2O MBDVC P5Z P62 PHGZM PHGZT PKEHL PQBIZ PQBZA PQEST PQGLB PQQKQ PQUKI Q9U 1XC
DOI	10.1007/s10766-022-00741-6
DatabaseName	SpringerOpen CrossRef ProQuest Central (Corporate) Computer and Information Systems Abstracts ABI/INFORM Collection ABI/INFORM Global (PDF only) ProQuest Central (purchase pre-March 2016) ABI/INFORM Global (Alumni Edition) Computing Database (Alumni Edition) Technology Research Database ProQuest SciTech Collection ProQuest Technology Collection ProQuest Central (Alumni) (purchase pre-March 2016) ABI/INFORM Collection (Alumni) Research Library (Alumni) ProQuest Central (Alumni) ProQuest Central UK/Ireland Advanced Technologies & Computer Science Collection ProQuest Central Essentials - QC ProQuest Central Business Premium Collection ProQuest Technology Collection ProQuest One ProQuest Central Korea Business Premium Collection (Alumni) ABI/INFORM Global (Corporate) ProQuest Central Student ProQuest Research Library SciTech Premium Collection ProQuest Computer Science Collection ProQuest Business Collection (Alumni Edition) ProQuest Business Collection Computer Science Database ABI/INFORM Professional Advanced Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional ABI/INFORM Global Computing Database Research Library Research Library (Corporate) Advanced Technologies & Aerospace Database ProQuest Advanced Technologies & Aerospace Collection ProQuest One Academic ProQuest One Academic (New) ProQuest One Academic Middle East (New) ProQuest One Business ProQuest One Business (Alumni) ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Applied & Life Sciences ProQuest One Academic (retired) ProQuest One Academic UKI Edition ProQuest Central Basic Hyper Article en Ligne (HAL)
DatabaseTitle	CrossRef ABI/INFORM Global (Corporate) ProQuest Business Collection (Alumni Edition) ProQuest One Business Research Library Prep Computer Science Database ProQuest Central Student Technology Collection Technology Research Database Computer and Information Systems Abstracts – Academic ProQuest One Academic Middle East (New) ProQuest Advanced Technologies & Aerospace Collection ProQuest Central Essentials ProQuest Computer Science Collection Computer and Information Systems Abstracts ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College Research Library (Alumni Edition) ABI/INFORM Complete ProQuest Central ABI/INFORM Professional Advanced ProQuest One Applied & Life Sciences ProQuest Central Korea ProQuest Research Library ProQuest Central (New) Advanced Technologies Database with Aerospace ABI/INFORM Complete (Alumni Edition) Advanced Technologies & Aerospace Collection Business Premium Collection ABI/INFORM Global ProQuest Computing ABI/INFORM Global (Alumni Edition) ProQuest Central Basic ProQuest Computing (Alumni Edition) ProQuest One Academic Eastern Edition ProQuest Technology Collection ProQuest SciTech Collection ProQuest Business Collection Computer and Information Systems Abstracts Professional Advanced Technologies & Aerospace Database ProQuest One Academic UKI Edition ProQuest One Business (Alumni) ProQuest One Academic ProQuest One Academic (New) ProQuest Central (Alumni) Business Premium Collection (Alumni)
DatabaseTitleList	ABI/INFORM Global (Corporate) CrossRef
Database_xml	– sequence: 1 dbid: BENPR name: ProQuest Central url: https://www.proquest.com/central sourceTypes: Aggregation Database
DeliveryMethod	fulltext_linktorsrc
Discipline	Computer Science
EISSN	1573-7640
EndPage	127
ExternalDocumentID	oai:HAL:hal-04551101v1 10_1007_s10766_022_00741_6
GroupedDBID	-4Z -59 -5G -BR -EM -Y2 -~C -~X .4S .86 .DC .VR 06D 0R~ 0VY 199 1N0 2.D 203 28- 29J 2J2 2JN 2JY 2KG 2LR 2P1 2VQ 2~H 30V 3V. 4.4 406 408 409 40D 40E 5GY 5QI 5VS 67Z 6NX 78A 7WY 8FE 8FG 8FL 8G5 8TC 8UJ 95- 95. 95~ 96X AAAVM AABHQ AACDK AAHNG AAIAL AAJBT AAJKR AANZL AAOBN AARHV AARTL AASML AATNV AATVU AAUYE AAWCG AAYIU AAYJJ AAYQN AAYTO AAYZH ABAKF ABBBX ABBXA ABDBF ABDPE ABDZT ABECU ABFSI ABFTD ABFTV ABHLI ABHQN ABJNI ABJOX ABKCH ABKTR ABMNI ABMQK ABNWP ABQBU ABQSL ABSXP ABTAH ABTEG ABTHY ABTKH ABTMW ABULA ABUWG ABWNU ABXPI ACAOD ACBXY ACDTI ACGFO ACGFS ACHSB ACHXU ACIHN ACKNC ACMDZ ACMLO ACNCT ACOKC ACOMO ACPIV ACREN ACUHS ACZOJ ADHIR ADINQ ADKNI ADKPE ADMLS ADRFC ADTPH ADURQ ADYFF ADYOE ADZKW AEAQA AEBTG AEFIE AEFQL AEGAL AEGNC AEJHL AEJRE AEKMD AEMSY AENEX AEOHA AEPYU AESKC AETLH AEVLU AEXYK AFBBN AFEXP AFGCZ AFKRA AFLOW AFQWF AFWTZ AFYQB AFZKB AGAYW AGDGC AGGDS AGJBK AGMZJ AGQEE AGQMX AGRTI AGWIL AGWZB AGYKE AHAVH AHBYD AHKAY AHSBF AHYZX AIAKS AIGIU AIIXL AILAN AITGF AJBLW AJRNO AJZVZ ALMA_UNASSIGNED_HOLDINGS ALWAN AMKLP AMTXH AMXSW AMYLF AOCGG ARAPS ARCSS ARMRJ AXYYD AYJHY AZFZN AZQEC B-. B0M BA0 BBWZM BDATZ BENPR BEZIV BGLVJ BGNMA BKOMP BPHCQ BSONS C6C CAG CCPQU COF CS3 CSCUP DDRTE DL5 DNIVK DPUIP DU5 DWQXO E.L EAD EAP EAS EBLON EBS EDO EIOEI EJD EMK EPL ESBYG ESX FEDTE FERAY FFXSO FIGPU FINBP FNLPD FRNLG FRRFC FSGXE FWDCC GGCAI GGRSB GJIRD GNUQQ GNWQR GQ6 GQ7 GQ8 GROUPED_ABI_INFORM_COMPLETE GROUPED_ABI_INFORM_RESEARCH GUQSH GXS H13 HCIFZ HF~ HG5 HG6 HMJXF HQYDN HRMNR HVGLF HZ~ H~9 I-F I09 IHE IJ- IKXTQ ITM IWAJR IXC IZIGR IZQ I~X I~Z J-C J0Z JBSCW JCJTX JZLTJ K60 K6V K6~ K7- KDC KOV KOW LAK LLZTM M0C M0N M2O M4Y MA- MS~ N2Q NB0 NDZJH NPVJJ NQJWS NU0 O9- O93 O9G O9I O9J OAM OVD P19 P62 P9O PF0 PQBIZ PQBZA PQQKQ PROAC PT4 PT5 Q2X QOK QOS R89 R9I RHV RNI RNS ROL RPX RSV RZC RZE RZK S16 S1Z S26 S27 S28 S3B SAP SCJ SCLPG SCO SDH SDM SHX SISQX SJYHP SNE SNPRN SNX SOHCF SOJ SPISZ SRMVM SSLCW STPWE SZN T13 T16 TAE TEORI TN5 TSG TSK TSV TUC TUS U2A U5U UG4 UOJIU UTJUX UZXMN VC2 VFIZW VXZ W23 W48 WH7 WK8 YLTOR Z45 Z7R Z7X Z81 Z83 Z88 Z8R Z8W Z92 ZMTXR ZY4 ~8M ~EX AAPKM AAYXX ABBRH ABDBE ABFSG ABRTQ ACSTC ADHKG AEZWR AFDZB AFFHD AFHIU AFOHR AGQPQ AHPBZ AHWEU AIXLP ATHPR AYFIA CITATION PHGZM PHGZT PQGLB 7SC 7XB 8AL 8FD 8FK JQ2 L.- L7M L~C L~D MBDVC PKEHL PQEST PQUKI Q9U 1XC
ID	FETCH-LOGICAL-c397t-91d1354b6b9ee7d2fb93067f2486d06336f75cfa7a3d1a937eed73701f9af5ea3
IEDL.DBID	RSV
ISICitedReferencesCount	1
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000897136200001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	0885-7458
IngestDate	Tue Oct 14 20:45:38 EDT 2025 Wed Nov 05 02:40:36 EST 2025 Tue Nov 18 22:38:59 EST 2025 Sat Nov 29 01:59:46 EST 2025 Fri Feb 21 02:43:33 EST 2025
IsDoiOpenAccess	true
IsOpenAccess	true
IsPeerReviewed	true
IsScholarly	true
Issue	2-3
Keywords	Parallel modeling Distributed training Deep network networks Performance analysis Functional transformation Symbolic cost model Distributed training ; Deep network networks; Parallel modeling; Symbolic cost model; Functional transformation; Performance analysis
Language	English
License	Distributed under a Creative Commons Attribution 4.0 International License: http://creativecommons.org/licenses/by/4.0
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c397t-91d1354b6b9ee7d2fb93067f2486d06336f75cfa7a3d1a937eed73701f9af5ea3
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
OpenAccessLink	https://link.springer.com/10.1007/s10766-022-00741-6
PQID	2792728052
PQPubID	48389
PageCount	19
ParticipantIDs	hal_primary_oai_HAL_hal_04551101v1 proquest_journals_2792728052 crossref_citationtrail_10_1007_s10766_022_00741_6 crossref_primary_10_1007_s10766_022_00741_6 springer_journals_10_1007_s10766_022_00741_6
PublicationCentury	2000
PublicationDate	2023-06-01
PublicationDateYYYYMMDD	2023-06-01
PublicationDate_xml	– month: 06 year: 2023 text: 2023-06-01 day: 01
PublicationDecade	2020
PublicationPlace	New York
PublicationPlace_xml	– name: New York
PublicationTitle	International journal of parallel programming
PublicationTitleAbbrev	Int J Parallel Prog
PublicationYear	2023
Publisher	Springer US Springer Nature B.V Springer Verlag
Publisher_xml	– name: Springer US – name: Springer Nature B.V – name: Springer Verlag
References	CR19 CR18 CR16 CR14 CR12 CR11 CR10 Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin, Gimelshein, Antiga (CR17) 2019; 32 Gibbons (CR22) 1996; 6 Valiant (CR20) 2011; 77 Morihata, Matsuzaki, Hu, Takeichi (CR21) 2009; 44 Huang, Cheng, Bapna, Firat, Chen, Chen, Lee, Ngiam, Le, Wu, Chen (CR6) 2019 CR2 Cai, Yan, Ma, Wu, Huang, Cheng, Su, Yu (CR8) 2022; 33 CR3 Brown, Mann, Ryder (CR1) 2020; 33 CR5 CR7 CR29 CR28 CR9 CR27 CR26 CR25 CR24 Raffel, Shazeer, Roberts, Lee, Narang, Matena, Zhou, Li, Liu (CR15) 2020; 21 Krizhevsky, Sutskever, Hinton (CR4) 2012; 60 Wang, Huang, Li (CR13) 2019 Huet (CR23) 1997; 7 741_CR24 741_CR25 741_CR26 741_CR27 T Brown (741_CR1) 2020; 33 741_CR3 741_CR2 741_CR5 741_CR7 LG Valiant (741_CR20) 2011; 77 741_CR9 C Raffel (741_CR15) 2020; 21 M Wang (741_CR13) 2019 G Huet (741_CR23) 1997; 7 741_CR18 741_CR19 741_CR14 741_CR16 741_CR10 741_CR11 741_CR12 A Morihata (741_CR21) 2009; 44 A Krizhevsky (741_CR4) 2012; 60 Z Cai (741_CR8) 2022; 33 741_CR28 Y Huang (741_CR6) 2019 741_CR29 A Paszke (741_CR17) 2019; 32 J Gibbons (741_CR22) 1996; 6
References_xml	– volume: 7 start-page: 549 issue: 5 year: 1997 end-page: 554 ident: CR23 article-title: The zipper. publication-title: J. Funct. Program. doi: 10.1017/S0956796897002864 – ident: CR18 – volume: 33 start-page: 1877 year: 2020 end-page: 1901 ident: CR1 article-title: Language models are few-shot learners publication-title: Adv. Neural Inf. Process. Syst. – ident: CR14 – ident: CR2 – ident: CR16 – ident: CR12 – volume: 44 start-page: 177 issue: 1 year: 2009 end-page: 185 ident: CR21 article-title: The third homomorphism theorem on trees: downward & upward lead to divide-and-conquer publication-title: SIGPLAN Not. doi: 10.1145/1594834.1480905 – ident: CR10 – ident: CR29 – volume: 77 start-page: 154 issue: 1 year: 2011 end-page: 166 ident: CR20 article-title: A bridging model for multi-core computing publication-title: J. Comput. Syst. Sci. doi: 10.1016/j.jcss.2010.06.012 – volume: 33 start-page: 1967 issue: 8 year: 2022 end-page: 1981 ident: CR8 article-title: Tensoropt: exploring the tradeoffs in distributed DNN training with auto-parallelism publication-title: IEEE Trans. Parallel. Distrib. Syst. doi: 10.1109/TPDS.2021.3132413 – ident: CR25 – ident: CR27 – ident: CR19 – volume: 21 start-page: 1 issue: 140 year: 2020 end-page: 67 ident: CR15 article-title: Exploring the limits of transfer learning with a unified text-to-text transformer publication-title: J. Mach. Learn. Res. – ident: CR3 – year: 2019 ident: CR6 publication-title: GPipe: Efficient training of giant neural networks using pipeline parallelism – volume: 6 start-page: 657 issue: 4 year: 1996 end-page: 665 ident: CR22 article-title: Functional pearls: the third homomorphism theorem publication-title: J. Funct. Program. doi: 10.1017/S0956796800001908 – ident: CR11 – ident: CR9 – year: 2019 ident: CR13 publication-title: Supporting Very Large Models using Automatic Dataflow Graph Partitioning. EuroSys ’19 – volume: 60 start-page: 1097 issue: 6 year: 2012 end-page: 1105 ident: CR4 article-title: Imagenet classification with deep convolutional neural networks publication-title: Adv. Neural Inf. Process. Syst. – ident: CR5 – ident: CR7 – volume: 32 start-page: 8026 year: 2019 end-page: 8037 ident: CR17 article-title: Pytorch: an imperative style, high-performance deep learning library publication-title: Adv. Neural Inf. Process. Syst. – ident: CR28 – ident: CR26 – ident: CR24 – volume: 21 start-page: 1 issue: 140 year: 2020 ident: 741_CR15 publication-title: J. Mach. Learn. Res. – ident: 741_CR26 – volume: 32 start-page: 8026 year: 2019 ident: 741_CR17 publication-title: Adv. Neural Inf. Process. Syst. – ident: 741_CR28 – volume: 33 start-page: 1967 issue: 8 year: 2022 ident: 741_CR8 publication-title: IEEE Trans. Parallel. Distrib. Syst. doi: 10.1109/TPDS.2021.3132413 – ident: 741_CR11 – ident: 741_CR19 doi: 10.1007/978-3-030-85665-6_13 – volume: 6 start-page: 657 issue: 4 year: 1996 ident: 741_CR22 publication-title: J. Funct. Program. doi: 10.1017/S0956796800001908 – ident: 741_CR24 – ident: 741_CR9 doi: 10.1145/3437801.3441593 – volume: 77 start-page: 154 issue: 1 year: 2011 ident: 741_CR20 publication-title: J. Comput. Syst. Sci. doi: 10.1016/j.jcss.2010.06.012 – ident: 741_CR29 – ident: 741_CR27 – volume: 7 start-page: 549 issue: 5 year: 1997 ident: 741_CR23 publication-title: J. Funct. Program. doi: 10.1017/S0956796897002864 – ident: 741_CR14 – ident: 741_CR12 – ident: 741_CR10 – volume-title: Supporting Very Large Models using Automatic Dataflow Graph Partitioning. EuroSys ’19 year: 2019 ident: 741_CR13 – ident: 741_CR18 – ident: 741_CR3 doi: 10.1145/2988450.2988454 – volume: 44 start-page: 177 issue: 1 year: 2009 ident: 741_CR21 publication-title: SIGPLAN Not. doi: 10.1145/1594834.1480905 – ident: 741_CR16 – ident: 741_CR2 doi: 10.1109/CVPR.2016.90 – volume-title: GPipe: Efficient training of giant neural networks using pipeline parallelism year: 2019 ident: 741_CR6 – ident: 741_CR7 – ident: 741_CR5 – ident: 741_CR25 – volume: 60 start-page: 1097 issue: 6 year: 2012 ident: 741_CR4 publication-title: Adv. Neural Inf. Process. Syst. – volume: 33 start-page: 1877 year: 2020 ident: 741_CR1 publication-title: Adv. Neural Inf. Process. Syst.
SSID	ssj0009788
Score	2.2974896
Snippet	The increasing size of deep neural networks (DNNs) raises a high demand for distributed training. An expert could find good hybrid parallelism strategies, but... Abstract The increasing size of deep neural networks (DNNs) raises a high demand for distributed training. An expert could find good hybrid parallelism...
SourceID	hal proquest crossref springer
SourceType	Open Access Repository Aggregation Database Enrichment Source Index Database Publisher
StartPage	109
SubjectTerms	Artificial neural networks Communication Computation Computer Science Cost analysis Cost function Homomorphisms Mathematical analysis Parallel processing Portability Processor Architectures Searching Semantics Software Engineering/Programming and Operating Systems Special Issue on High-Level Parallel Programming and Applications (HLPP 2022) Theory of Computation Training
SummonAdditionalLinks	– databaseName: ABI/INFORM Global dbid: M0C link: http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1dT9swFL2isIe9ANuYVj4ma-JtWMRxEje8IETXMQmqShSpb9FNbGtI0EJb-P3cmzoUJsHLXmPnQzm277F9fQ7AfqUrj2lJcxMTJTJBjRIxU5Kiq68wdp3cYm02Yfr9zmiUD8KC2yykVTZjYj1Q20nFa-SHLHTHVkppfHx3L9k1indXg4VGC9aY2XBK30V0uhTdNbXvJHWkVJok7YRDM-HonMk4_TaWdVSV2avA1PrLaZEvOOc_26R19Olt_O93b8J64J3iZNFQPsGKG3-GjcbTQYQu_gX-XF5c_j4Sg9rKmz5F9qbOiQFO2XPl5np2K9g9jUsE0V3RZd1dtsxyVgyD24SYeNHt97fgqvdreHomg92CrIiUzGnYs0qnSZmVuXPGxr7MeT7h46STWWIyOvMmJWQNaquQaA2FV6NNpHyOPnWov8LqeDJ230B4pcqMaLu2Nkkqk-QlloiRdo6mM4i-Dar510UVtMjZEuOmWKooMz4F4VPU-BRZG34-33O3UOJ4t_YPgvC5Ioton52cF3yNSCyxzEg9qjbsNpgVoevOiiVgbThoUF8Wv_3K7feftgMf2ap-kWa2C6vz6YPbgw_V4_x6Nv1eN9wnvrLwag priority: 102 providerName: ProQuest
Title	SMSG: Profiling-Free Parallelism Modeling for Distributed Training of DNN
URI	https://link.springer.com/article/10.1007/s10766-022-00741-6 https://www.proquest.com/docview/2792728052 https://univ-orleans.hal.science/hal-04551101
Volume	51
WOSCitedRecordID	wos000897136200001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVAVX databaseName: SpringerLINK Contemporary 1997-Present customDbUrl: eissn: 1573-7640 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0009788 issn: 0885-7458 databaseCode: RSV dateStart: 19970101 isFulltext: true titleUrlDefault: https://link.springer.com/search?facet-content-type=%22Journal%22 providerName: Springer Nature
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1NT9wwEB11oQcu0BYQ29KVVfVWLMVxEie9UWBL1ZJGLN-XaJLYAgmWanfL7-9MNmEpAiS4WErsfGjG9jzL4_cAPpe6dBgWtDYxXiAD1CgRIyUpuroSfRsnFdZiEyZN45OTJGsOhY3bbPd2S7Keqe8cdjMRJ8z6so6DMurAPIW7mAUb9gdHM6pdU6tN0vAJpQnCuDkq8_A7_gtHnXNOhryDNO9tjtYxp7_0sr99A4sNxhSb007xFl7Z4TtYavUbRDOcl-HHYG_w_avIatluernsj6wVGY5YX-XyYnwlWCmNawRBW7HNHLssj2UrcdAoS4hrJ7bTdAUO-zsHW7uykVaQJQGQCU1xldJhUERFYq2pfFckvHZwfhBHFaEWHTkTkhcN6kohQRgKpUYbT7kEXWhRr8Lc8Hpo10A4pYqIILquqiAoTZAUWCB62lpauiC6LqjWwnnZ8I6z_MVlPmNMZlvlZKu8tlUedeHL7TN_pqwbT7b-RI67bciE2bubv3K-R4CVEKWnblQX1lu_5s0wHefMnsj6XKHfhY3Wj7Pqxz_5_nnNP8ACy9RPU8zWYW4y-ms_wuvyZnIxHvWgY45PezD_bSfN9unqp5FU7nlbXPq_qczCs17dyf8BtKHswg
linkProvider	Springer Nature
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V3fb9MwED5tAwleGOOHKAywJngCizp24gQJoYmua9UuqrQi7c04sS0mjXa0ZdP-Kf5G7tJk3ZDY2x54jRNHjr-77xyf7wN4U8oy2LjAtYluK66stNzaRHBk11DayKeZs5XYhM7z9OgoG63B7-YsDKVVNj6xctRuWtI_8g9U6I6klOLo8-lPTqpRtLvaSGgsYTHwF-e4ZJt_6ndwft9GUXdv_KXHa1UBXiL3LtC6nZCxKpIi8167KBQZhc0hUmnikLBlEnSMA9BWOmGRvZFFtNRtETIbYm8l9rsOd5RMNdnVQPNVkV9d6Vyi4cZcqzitD-nUR_V0Qum-Ea9YnCfXiHD9O6VhXolx_9qWrdiuu_m_faeH8KCOq9nu0hC2YM1PHsFmo1nBahf2GPqHB4f7H9mokirHofPuzHs2sjPSlDk5nv9gpA5HLQzDedahusIkCeYdG9dqGmwaWCfPn8DXWxnQU9iYTCf-GbAgRJHgskQ6p1SpVVbYwtq29B6Xa9aGFohmbk1Z11onyY8Ts6oSTXgwiAdT4cEkLXh3-czpstLIjXfvIGQub6Qi4b3doaFrGKRjFN0WZ6IF2w1GTO2a5mYFkBa8b1C2av73K5_f3NtruNcbHwzNsJ8PXsD9CIPBZUrdNmwsZr_8S7hbni2O57NXldEw-Hbb6PsD2BdN2A
linkToPdf	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V1bT9swFD4CNk28wK5agW3WtD1tFnWcxM0kNKGVjgoWRYJJaC_GiW2BBC20BcRf26_jnNShbNJ442GvsePI9ndu8fH5AD5UsvImKTE2Ue2Yx0YabkwqOFpXX5nIdTJrarIJleedg4OsmIPfzV0YSqtsdGKtqO2won_k61TojqiUkmjdh7SIotv7enbOiUGKTlobOo0pRHbc9RWGb-ONfhf3-mMU9bb2v23zwDDAK7TDE5R0K2QSl2mZOads5MuMXGgfxZ3UovGWqVcJTkYZaYVBS44WRUnVFj4zPnFG4rjz8EhhjEnphEXya1bwV9WclyjECVdx0gkXdsK1PZVS6m_Ea4vO0z-M4vwRpWTe8Xf_OqKtLV9v-X9es6ewFPxttjkVkGcw5wbPYbnhsmBBtb2A_t6Pve9fWFFTmOMy8N7IOVaYEXHNnByPTxmxxlELQzefdaneMFGFOcv2A8sGG3rWzfOX8PNBJvQKFgbDgXsNzAtRphiuSGvjuFJxVprSmLZ0DsM4Y3wLRLPPugo12IkK5ETPqkcTNjRiQ9fY0GkLPt2-czatQHJv7_cIn9uOVDx8e3NX0zN03tG7botL0YK1Bi86qKyxnoGlBZ8bxM2a__3JlftHewdPEHR6t5_vrMJihD7iNNNuDRYmowv3Bh5Xl5Pj8ehtLT8MDh8afDe-UVb8
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=SMSG%3A+Profiling-Free+Parallelism+Modeling+for+Distributed+Training+of+DNN&rft.jtitle=International+journal+of+parallel+programming&rft.au=Wang%2C+Haoran&rft.au=Tachon%2C+Thibaut&rft.au=Li%2C+Chong&rft.au=Robert%2C+Sophie&rft.date=2023-06-01&rft.pub=Springer+US&rft.issn=0885-7458&rft.eissn=1573-7640&rft.volume=51&rft.issue=2-3&rft.spage=109&rft.epage=127&rft_id=info:doi/10.1007%2Fs10766-022-00741-6&rft.externalDocID=10_1007_s10766_022_00741_6
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0885-7458&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0885-7458&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0885-7458&client=summon