A high performance implementation of MPI-IO for a Lustre file system environment
It is often the case that MPI‐IO performs poorly in a Lustre file system environment, although the reasons for such performance have heretofore not been well understood. We hypothesize that such performance is a direct result of the fundamental assumptions upon which most parallel I/O optimizations...
Uložené v:
| Vydané v: | Concurrency and computation Ročník 22; číslo 11; s. 1433 - 1449 |
|---|---|
| Hlavní autori: | , |
| Médium: | Journal Article |
| Jazyk: | English |
| Vydavateľské údaje: |
Chichester, UK
John Wiley & Sons, Ltd
10.08.2010
|
| Predmet: | |
| ISSN: | 1532-0626, 1532-0634, 1532-0634 |
| On-line prístup: | Získať plný text |
| Tagy: |
Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
|
| Abstract | It is often the case that MPI‐IO performs poorly in a Lustre file system environment, although the reasons for such performance have heretofore not been well understood. We hypothesize that such performance is a direct result of the fundamental assumptions upon which most parallel I/O optimizations are based. In particular, it is almost universally believed that parallel I/O performance is optimized when aggregator processes perform large, contiguous I/O operations in parallel. Our research, however, shows that this approach can actually provide the worst performance in a Lustre environment, and that the best performance may be obtained by performing a large number of small, non‐contiguous I/O operations. In this paper, we provide empirical results demonstrating these non‐intuitive results and explore the reasons for such unexpected performance. We present our solution to the problem, which is embodied in a user‐level library termed Y‐Lib, which redistributes the data in a way that conforms much more closely with the Lustre storage architecture than does the data redistribution pattern employed by MPI‐IO. We provide a large body of experimental results, taken across two large‐scale Lustre installations, demonstrating that Y‐Lib outperforms MPI‐IO by up to 36% on one system and 1000% on the other. We discuss the factors that impact the performance improvement obtained by Y‐Lib, which include the number of aggregator processes and Object Storage Devices, as well as the power of the system's communications infrastructure. We also show that the optimal data redistribution pattern for Y‐Lib is dependent upon these same factors. Copyright © 2009 John Wiley & Sons, Ltd. |
|---|---|
| AbstractList | It is often the case that MPI‐IO performs poorly in a Lustre file system environment, although the reasons for such performance have heretofore not been well understood. We hypothesize that such performance is a direct result of the fundamental assumptions upon which most parallel I/O optimizations are based. In particular, it is almost universally believed that parallel I/O performance is optimized when aggregator processes perform large, contiguous I/O operations in parallel. Our research, however, shows that this approach can actually provide the worst performance in a Lustre environment, and that the best performance may be obtained by performing a large number of small, non‐contiguous I/O operations. In this paper, we provide empirical results demonstrating these non‐intuitive results and explore the reasons for such unexpected performance. We present our solution to the problem, which is embodied in a user‐level library termed Y‐Lib, which redistributes the data in a way that conforms much more closely with the Lustre storage architecture than does the data redistribution pattern employed by MPI‐IO. We provide a large body of experimental results, taken across two large‐scale Lustre installations, demonstrating that Y‐Lib outperforms MPI‐IO by up to 36% on one system and 1000% on the other. We discuss the factors that impact the performance improvement obtained by Y‐Lib, which include the number of aggregator processes and Object Storage Devices, as well as the power of the system's communications infrastructure. We also show that the optimal data redistribution pattern for Y‐Lib is dependent upon these same factors. Copyright © 2009 John Wiley & Sons, Ltd. It is often the case that MPI-IO performs poorly in a Lustre file system environment, although the reasons for such performance have heretofore not been well understood. We hypothesize that such performance is a direct result of the fundamental assumptions upon which most parallel I/O optimizations are based. In particular, it is almost universally believed that parallel I/O performance is optimized when aggregator processes perform large, contiguous I/O operations in parallel. Our research, however, shows that this approach can actually provide the worst performance in a Lustre environment, and that the best performance may be obtained by performing a large number of small, non-contiguous I/O operations. In this paper, we provide empirical results demonstrating these non-intuitive results and explore the reasons for such unexpected performance. We present our solution to the problem, which is embodied in a user-level library termed Y-Lib, which redistributes the data in a way that conforms much more closely with the Lustre storage architecture than does the data redistribution pattern employed by MPI-IO. We provide a large body of experimental results, taken across two large-scale Lustre installations, demonstrating that Y-Lib outperforms MPI-IO by up to 36% on one system and 1000% on the other. We discuss the factors that impact the performance improvement obtained by Y-Lib, which include the number of aggregator processes and Object Storage Devices, as well as the power of the system's communications infrastructure. We also show that the optimal data redistribution pattern for Y-Lib is dependent upon these same factors. |
| Author | Dickens, Phillip M. Logan, Jeremy |
| Author_xml | – sequence: 1 givenname: Phillip M. surname: Dickens fullname: Dickens, Phillip M. email: dickens@umcs.maine.edu organization: Department of Computer Science, University of Maine, Orono, ME, U.S.A – sequence: 2 givenname: Jeremy surname: Logan fullname: Logan, Jeremy organization: Department of Computer Science, University of Maine, Orono, ME, U.S.A |
| BookMark | eNp1kEFPwjAYhhuDiYAm_oQevQzbde3YkQxEEhQOKsemlG9S3dbZDpV_78gMxkRP33d43jd5nx7qlLYEhC4pGVBCwmtdwYBGCT1BXcpZGBDBos7xD8UZ6nn_QgilhNEuWo7w1jxvcQUus65QpQZsiiqHAspa1caW2Gb4bjkLZgvcEFjh-c7XDnBmcsB-72soMJTvxtnykDlHp5nKPVx83z56vJk8pLfBfDGdpaN5oBkTNIBwE64ZI8k6TIZKK7URjEdcRaIZoWPQEA5VxiGm0SaLEkUSYBEIHSrKiOAJ66Ortrdy9m0HvpaF8RryXJVgd15SEVM25JyLBh20qHbWeweZ1KbdVjtlckmJPKiTjTp5UPfTfQxUzhTK7f9Cgxb9aHTs_-Vkupz85k0j7vPIK_cqRcxiLlf3UzmOV2PCnhKZsi_afo0w |
| CitedBy_id | crossref_primary_10_1109_ACCESS_2020_2985928 crossref_primary_10_1016_j_cpc_2024_109246 crossref_primary_10_1016_j_parco_2021_102786 crossref_primary_10_1002_mp_13033 crossref_primary_10_1007_s11227_011_0669_2 crossref_primary_10_1002_cpe_1613 |
| Cites_doi | 10.1109/CCGRID.2007.51 10.1016/S0167-8191(01)00129-6 10.1109/CLUSTR.2003.1253332 10.1109/IPDPS.2007.370485 10.1145/301816.301826 10.1109/FMPC.1999.750599 10.1109/IPDPS.2007.370239 |
| ContentType | Journal Article |
| Copyright | Copyright © 2009 John Wiley & Sons, Ltd. |
| Copyright_xml | – notice: Copyright © 2009 John Wiley & Sons, Ltd. |
| DBID | BSCLL AAYXX CITATION 7SC 8FD H8D JQ2 L7M L~C L~D |
| DOI | 10.1002/cpe.1491 |
| DatabaseName | Istex CrossRef Computer and Information Systems Abstracts Technology Research Database Aerospace Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
| DatabaseTitle | CrossRef Aerospace Database Technology Research Database Computer and Information Systems Abstracts – Academic ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Aerospace Database CrossRef |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISSN | 1532-0634 |
| EndPage | 1449 |
| ExternalDocumentID | 10_1002_cpe_1491 CPE1491 ark_67375_WNG_D7WD03V9_C |
| Genre | article |
| GrantInformation_xml | – fundername: National Science Foundation funderid: 0702748 |
| GroupedDBID | .3N .DC .GA .Y3 05W 0R~ 10A 1L6 1OC 31~ 33P 3SF 3WU 4.4 50Y 50Z 51W 51X 52M 52N 52O 52P 52S 52T 52U 52W 52X 5GY 5VS 66C 702 7PT 8-0 8-1 8-3 8-4 8-5 8UM 930 A03 AAESR AAEVG AAHQN AAMNL AANHP AANLZ AAONW AASGY AAXRX AAYCA AAZKR ABCQN ABCUV ABEML ABIJN ACAHQ ACBWZ ACCZN ACPOU ACRPL ACSCC ACXBN ACXQS ACYXJ ADBBV ADEOM ADIZJ ADKYN ADMGS ADMLS ADNMO ADOZA ADXAS ADZMN AEIGN AEIMD AEUYR AEYWJ AFBPY AFFPM AFGKR AFWVQ AFZJQ AGQPQ AGYGG AHBTC AITYG AIURR AJXKR ALMA_UNASSIGNED_HOLDINGS ALVPJ AMBMR AMYDB ASPBG ATUGU AUFTA AVWKF AZBYB AZFZN BAFTC BDRZF BFHJK BHBCM BMNLL BROTX BRXPI BSCLL BY8 CS3 D-E D-F DCZOG DPXWK DR2 DRFUL DRSTM EBS EJD F00 F01 F04 F5P FEDTE G-S G.N GNP GODZA HF~ HGLYW HHY HVGLF HZ~ IX1 JPC KQQ LATKE LAW LC2 LC3 LEEKS LH4 LITHE LOXES LP6 LP7 LUTES LW6 LYRES MEWTI MK4 MRFUL MRSTM MSFUL MSSTM MXFUL MXSTM N04 N05 N9A O66 O9- OIG P2W P2X P4D PQQKQ Q.N Q11 QB0 QRW R.K ROL RX1 SUPJJ TN5 UB1 V2E W8V W99 WBKPD WIH WIK WOHZO WQJ WXSBR WYISQ WZISG XG1 XV2 ~IA ~WT AAHHS ACCFJ ADZOD AEEZP AEQDE AEUQT AFPWT AIWBW AJBDE ALUQN RWI WRC AAYXX CITATION O8X 7SC 8FD H8D JQ2 L7M L~C L~D |
| ID | FETCH-LOGICAL-c3361-e2d2b3309b298acaad63545a46002c7ece28af5e714df49a09e34e6c2a1306593 |
| IEDL.DBID | DRFUL |
| ISICitedReferencesCount | 10 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000280431500006&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1532-0626 1532-0634 |
| IngestDate | Sun Nov 09 09:26:25 EST 2025 Sat Nov 29 01:41:10 EST 2025 Tue Nov 18 21:16:21 EST 2025 Wed Jan 22 16:19:40 EST 2025 Tue Nov 11 03:32:29 EST 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 11 |
| Language | English |
| License | http://onlinelibrary.wiley.com/termsAndConditions#vor |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c3361-e2d2b3309b298acaad63545a46002c7ece28af5e714df49a09e34e6c2a1306593 |
| Notes | National Science Foundation - No. 0702748 ark:/67375/WNG-D7WD03V9-C ArticleID:CPE1491 istex:51734B87519EF8407F97C21154D37ED12BBCD575 ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 |
| PQID | 1671385556 |
| PQPubID | 23500 |
| PageCount | 17 |
| ParticipantIDs | proquest_miscellaneous_1671385556 crossref_citationtrail_10_1002_cpe_1491 crossref_primary_10_1002_cpe_1491 wiley_primary_10_1002_cpe_1491_CPE1491 istex_primary_ark_67375_WNG_D7WD03V9_C |
| PublicationCentury | 2000 |
| PublicationDate | 2010-08-10 10 August 2010 20100810 |
| PublicationDateYYYYMMDD | 2010-08-10 |
| PublicationDate_xml | – month: 08 year: 2010 text: 2010-08-10 day: 10 |
| PublicationDecade | 2010 |
| PublicationPlace | Chichester, UK |
| PublicationPlace_xml | – name: Chichester, UK |
| PublicationTitle | Concurrency and computation |
| PublicationTitleAlternate | Concurrency Computat.: Pract. Exper |
| PublicationYear | 2010 |
| Publisher | John Wiley & Sons, Ltd |
| Publisher_xml | – name: John Wiley & Sons, Ltd |
| References | Thakur R, Gropp W, Lusk E. Optimizing noncontiguous accesses in MPI-IO. Parallel Computing 2002; 28(1):83-105. 2009 2002; 28 2007 2004 e_1_2_9_10_2 e_1_2_9_21_2 e_1_2_9_20_2 e_1_2_9_12_2 e_1_2_9_11_2 e_1_2_9_22_2 e_1_2_9_7_2 e_1_2_9_6_2 e_1_2_9_5_2 e_1_2_9_4_2 e_1_2_9_3_2 e_1_2_9_2_2 e_1_2_9_9_2 e_1_2_9_8_2 e_1_2_9_14_2 e_1_2_9_13_2 e_1_2_9_16_2 e_1_2_9_15_2 e_1_2_9_18_2 e_1_2_9_17_2 e_1_2_9_19_2 |
| References_xml | – reference: Thakur R, Gropp W, Lusk E. Optimizing noncontiguous accesses in MPI-IO. Parallel Computing 2002; 28(1):83-105. – year: 2009 – start-page: 23 end-page: 32 – volume: 28 start-page: 83 issue: 1 year: 2002 end-page: 105 article-title: Optimizing noncontiguous accesses in MPI‐IO publication-title: Parallel Computing – start-page: 104 end-page: 111 – start-page: 182 end-page: 189 – year: 2007 – year: 2004 – ident: e_1_2_9_17_2 – ident: e_1_2_9_21_2 – ident: e_1_2_9_8_2 doi: 10.1109/CCGRID.2007.51 – ident: e_1_2_9_14_2 doi: 10.1016/S0167-8191(01)00129-6 – ident: e_1_2_9_5_2 – ident: e_1_2_9_11_2 – ident: e_1_2_9_2_2 – ident: e_1_2_9_13_2 doi: 10.1109/CLUSTR.2003.1253332 – ident: e_1_2_9_6_2 – ident: e_1_2_9_19_2 doi: 10.1109/IPDPS.2007.370485 – ident: e_1_2_9_3_2 – ident: e_1_2_9_22_2 – ident: e_1_2_9_20_2 – ident: e_1_2_9_12_2 – ident: e_1_2_9_7_2 – ident: e_1_2_9_4_2 – ident: e_1_2_9_15_2 – ident: e_1_2_9_10_2 doi: 10.1145/301816.301826 – ident: e_1_2_9_9_2 doi: 10.1109/FMPC.1999.750599 – ident: e_1_2_9_16_2 – ident: e_1_2_9_18_2 doi: 10.1109/IPDPS.2007.370239 |
| SSID | ssj0011031 |
| Score | 1.9941487 |
| Snippet | It is often the case that MPI‐IO performs poorly in a Lustre file system environment, although the reasons for such performance have heretofore not been well... It is often the case that MPI-IO performs poorly in a Lustre file system environment, although the reasons for such performance have heretofore not been well... |
| SourceID | proquest crossref wiley istex |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 1433 |
| SubjectTerms | Concurrency Devices grid computing Infrastructure Libraries Lustre Mathematical models object-based file systems Optimization parallel I/O Yttrium |
| Title | A high performance implementation of MPI-IO for a Lustre file system environment |
| URI | https://api.istex.fr/ark:/67375/WNG-D7WD03V9-C/fulltext.pdf https://onlinelibrary.wiley.com/doi/abs/10.1002%2Fcpe.1491 https://www.proquest.com/docview/1671385556 |
| Volume | 22 |
| WOSCitedRecordID | wos000280431500006&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVWIB databaseName: Wiley Online Library Full Collection 2020 customDbUrl: eissn: 1532-0634 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0011031 issn: 1532-0626 databaseCode: DRFUL dateStart: 20010101 isFulltext: true titleUrlDefault: https://onlinelibrary.wiley.com providerName: Wiley-Blackwell |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1LSwMxEA5iPXixPrG-iCB6WtxNNt3NUVqrQq1FfN1CNjuBoralreLRn-Bv9JeY2UdVUBA87WWSXWYymy-ZmW8I2TMp4wIg8AD81AultZ6OfesJHYtUM21tVl9x0446nfjuTnaLrEqshcn5IaYXbugZ2f8aHVwn48NP0lAzBOfmWLhewZoqd_CqNC9b1-1pDAEbGORsqczzHW4vqWd9dliO_bYZVVCvL9-Q5le8mm04rep_PnWRLBQwkx7l62KJzEB_mVTLFg608OgVcnlEkbCYDj_rB2jvscwpR6PRgaXn3bP317ezC-pkqKZtvB8BipRONGeCpl_q5VbJdev4qnHqFW0WPMN53VmJpSzh3JcJk7E2WqcOhIRChxiyMxEYYLG2AqIgTG0otS-Bh1A3TAfYdl7yNTLbH_RhnVCTCGGETACEDXnq4AZwN9xYhxJ1ECU1clDqW5mCgxxbYTyonD2ZKacqhaqqkd2p5DDn3fhBZj8z2VRAj-4xTy0S6rZzoprRbdPnN1I13GSlTZXzHgyJ6D4MnsYqqLtDeiyEqLvJMhP--jbV6B7jc-OvgptkPk81QP7cLTI7GT3BNpkzz5PeeLRTrNYP2CXu4w |
| linkProvider | Wiley-Blackwell |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1La9wwEB7CbqC9NEkfNG0eKpT2ZGJLlm2RU9jNJkud7RLyuglZHkFou7tsktJjf0J-Y35JNH5sEmih0JMvI9nMaKTPmplvAD7akguJGAWIYRnEyrnAZKELpMlkabhxrqqvOMvT0Si7uFDjJdhta2FqfojFhRt5RrVfk4PThfTOA2uonaH3c6pc78aJSLMOdPvHg9N8EUSgDgY1XSoPQg_cW-7ZkO-0Y5-cRl1S7K8nUPMxYK1OnMHKf33rKrxogCbbq1fGGizh5CWstE0cWOPTr-B4jxFlMZs9VBCwyx9tVjmZjU0dOxoP737fDr8yL8MMy-mGBBmROrGaC5o9qph7DaeD_ZPeYdA0WgisEIm3Ey95IUSoCq4yY40pPQyJpYkpaGdTtMgz4ySmUVy6WJlQoYgxsdxE1HheiTfQmUwn-BaYLaS0UhWI0sWi9IADhR9unceJJkqLdfjcKlzbhoWcmmF81zV_MtdeVZpUtQ4fFpKzmnnjDzKfKpstBMz8G2WqpVKfjw50Pz3vh-JM6Z6frDWq9v5DQREzwenNlY4S_5ueSSkTP1llw7--TffG-_R896-C2_Ds8OQo1_lw9OU9PK8TD4hNdwM61_Mb3IRl-_P68mq-1SzdeyLJ8tM |
| linkToPdf | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1LTxsxEB6hpKp6AUpbQQvUlVB7WrHrR3YtTighNGoIESqPm-X1jiXUkkQBqh77E_ob-SV49hFAaqVKPe1l7F3NeOxvPTPfAOy4gguFmESIcRFJ7X1ks9hHymaqsNx6X9ZXnA3T0Si7uNDjJdhramEqfojFhRt5Rrlfk4PjrPC7D6yhbobBz6lyvS2VVrIF7d5J_3S4CCJQB4OKLpVHcQDuDfdszHebsU9OozYp9ucTqPkYsJYnTn_lv751FZZroMn2q5XxEpZwsgYrTRMHVvv0KzjZZ0RZzGYPFQTs8qrJKiezsalnR-PB3a_fg2MWZJhlQ7ohQUakTqzigmaPKuZew2n_4Gv3c1Q3WoicEJ1gJ17wXIhY51xn1llbBBgilZUUtHMpOuSZ9QrTRBZeahtrFBI7jtuEGs9r8QZak-kE14G5XCmndI6ovBRFABwownDnA060SZpvwKdG4cbVLOTUDOO7qfiTuQmqMqSqDfiwkJxVzBt_kPlY2mwhYOffKFMtVeZ8dGh66XkvFmfadMNkjVFN8B8KitgJTm-vTdIJv-mZUqoTJitt-Ne3me74gJ5v_1XwPTwf9_pmOBh9eQcvqrwDItPdhNbN_Ba34Jn7cXN5Pd-uV-49wl3yTg |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+high+performance+implementation+of+MPI%E2%80%90IO+for+a+Lustre+file+system+environment&rft.jtitle=Concurrency+and+computation&rft.au=Dickens%2C+Phillip+M.&rft.au=Logan%2C+Jeremy&rft.date=2010-08-10&rft.issn=1532-0626&rft.eissn=1532-0634&rft.volume=22&rft.issue=11&rft.spage=1433&rft.epage=1449&rft_id=info:doi/10.1002%2Fcpe.1491&rft.externalDBID=n%2Fa&rft.externalDocID=10_1002_cpe_1491 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1532-0626&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1532-0626&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1532-0626&client=summon |