Parsl: Pervasive Parallel Programming in Python

High-level programming languages such as Python are increasingly used to provide intuitive interfaces to libraries written in lower-level languages and for assembling applications from various components. This migration towards orchestration rather than implementation, coupled with the growing need...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:arXiv.org
Hlavní autoři: Babuji, Yadu, Woodard, Anna, Li, Zhuozhao, Katz, Daniel S, Clifford, Ben, Kumar, Rohan, Lacinski, Lukasz, Chard, Ryan, Wozniak, Justin M, Foster, Ian, Wilde, Michael, Chard, Kyle
Médium: Paper
Jazyk:angličtina
Vydáno: Ithaca Cornell University Library, arXiv.org 18.05.2019
Témata:
Cosmology
Data management
Dependence
Distance learning
Fault tolerance
Machine learning
Materials science
Migration
Moore's law
Parallel programming
Programming languages
Provisioning
Python
ISSN:2331-8422
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Abstract High-level programming languages such as Python are increasingly used to provide intuitive interfaces to libraries written in lower-level languages and for assembling applications from various components. This migration towards orchestration rather than implementation, coupled with the growing need for parallel computing (e.g., due to big data and the end of Moore's law), necessitates rethinking how parallelism is expressed in programs. Here, we present Parsl, a parallel scripting library that augments Python with simple, scalable, and flexible constructs for encoding parallelism. These constructs allow Parsl to construct a dynamic dependency graph of components that it can then execute efficiently on one or many processors. Parsl is designed for scalability, with an extensible set of executors tailored to different use cases, such as low-latency, high-throughput, or extreme-scale execution. We show, via experiments on the Blue Waters supercomputer, that Parsl executors can allow Python scripts to execute components with as little as 5 ms of overhead, scale to more than 250 000 workers across more than 8000 nodes, and process upward of 1200 tasks per second. Other Parsl features simplify the construction and execution of composite programs by supporting elastic provisioning and scaling of infrastructure, fault-tolerant execution, and integrated wide-area data management. We show that these capabilities satisfy the needs of many-task, interactive, online, and machine learning applications in fields such as biology, cosmology, and materials science.
AbstractList High-level programming languages such as Python are increasingly used to provide intuitive interfaces to libraries written in lower-level languages and for assembling applications from various components. This migration towards orchestration rather than implementation, coupled with the growing need for parallel computing (e.g., due to big data and the end of Moore's law), necessitates rethinking how parallelism is expressed in programs. Here, we present Parsl, a parallel scripting library that augments Python with simple, scalable, and flexible constructs for encoding parallelism. These constructs allow Parsl to construct a dynamic dependency graph of components that it can then execute efficiently on one or many processors. Parsl is designed for scalability, with an extensible set of executors tailored to different use cases, such as low-latency, high-throughput, or extreme-scale execution. We show, via experiments on the Blue Waters supercomputer, that Parsl executors can allow Python scripts to execute components with as little as 5 ms of overhead, scale to more than 250 000 workers across more than 8000 nodes, and process upward of 1200 tasks per second. Other Parsl features simplify the construction and execution of composite programs by supporting elastic provisioning and scaling of infrastructure, fault-tolerant execution, and integrated wide-area data management. We show that these capabilities satisfy the needs of many-task, interactive, online, and machine learning applications in fields such as biology, cosmology, and materials science.
Author Li, Zhuozhao
Wozniak, Justin M
Katz, Daniel S
Chard, Ryan
Woodard, Anna
Chard, Kyle
Babuji, Yadu
Foster, Ian
Clifford, Ben
Kumar, Rohan
Lacinski, Lukasz
Wilde, Michael
Author_xml – sequence: 1
  givenname: Yadu
  surname: Babuji
  fullname: Babuji, Yadu
– sequence: 2
  givenname: Anna
  surname: Woodard
  fullname: Woodard, Anna
– sequence: 3
  givenname: Zhuozhao
  surname: Li
  fullname: Li, Zhuozhao
– sequence: 4
  givenname: Daniel
  surname: Katz
  middlename: S
  fullname: Katz, Daniel S
– sequence: 5
  givenname: Ben
  surname: Clifford
  fullname: Clifford, Ben
– sequence: 6
  givenname: Rohan
  surname: Kumar
  fullname: Kumar, Rohan
– sequence: 7
  givenname: Lukasz
  surname: Lacinski
  fullname: Lacinski, Lukasz
– sequence: 8
  givenname: Ryan
  surname: Chard
  fullname: Chard, Ryan
– sequence: 9
  givenname: Justin
  surname: Wozniak
  middlename: M
  fullname: Wozniak, Justin M
– sequence: 10
  givenname: Ian
  surname: Foster
  fullname: Foster, Ian
– sequence: 11
  givenname: Michael
  surname: Wilde
  fullname: Wilde, Michael
– sequence: 12
  givenname: Kyle
  surname: Chard
  fullname: Chard, Kyle
BookMark eNotjVFLwzAURoMoOOd-gG8Fn9slN71p4psMncLAPux93LbX2dElmmxF_70FffrgcDjfjbj0wbMQd0oWpUWUS4rf_VgoJ7GQoNBeiBlorXJbAlyLRUoHKSWYChD1TCxriml4yGqOI6V-5GwCNAw8ZHUM-0jHY-_3We-z-uf0EfytuHqnIfHif-di-_y0Xb3km7f16-pxkxOCyzU02BquWDrFmsDpEtnZrrFda1VL7BQCkMaGAKpGQ2ccSWdaxIonR8_F_V_2M4avM6fT7hDO0U-POwBQShpUTv8CUS5F3A
ContentType Paper
Copyright 2019. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Copyright_xml – notice: 2019. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
DBID 8FE
8FG
ABJCF
ABUWG
AFKRA
AZQEC
BENPR
BGLVJ
CCPQU
DWQXO
HCIFZ
L6V
M7S
PHGZM
PHGZT
PIMPY
PKEHL
PQEST
PQGLB
PQQKQ
PQUKI
PRINS
PTHSS
DOI 10.48550/arxiv.1905.02158
DatabaseName ProQuest SciTech Collection
ProQuest Technology Collection
Materials Science & Engineering Collection
ProQuest Central (Alumni)
ProQuest Central UK/Ireland
ProQuest Central Essentials
ProQuest Central Database Suite (ProQuest)
Technology Collection
ProQuest One
ProQuest Central Korea
SciTech Collection (ProQuest)
ProQuest Engineering Collection
Engineering Database
ProQuest Central Premium
ProQuest One Academic (New)
Publicly Available Content Database
ProQuest One Academic Middle East (New)
ProQuest One Academic Eastern Edition (DO NOT USE)
ProQuest One Applied & Life Sciences
ProQuest One Academic (retired)
ProQuest One Academic UKI Edition
ProQuest Central China
Engineering Collection
DatabaseTitle Publicly Available Content Database
Engineering Database
Technology Collection
ProQuest One Academic Middle East (New)
ProQuest Central Essentials
ProQuest One Academic Eastern Edition
ProQuest Central (Alumni Edition)
SciTech Premium Collection
ProQuest One Community College
ProQuest Technology Collection
ProQuest SciTech Collection
ProQuest Central China
ProQuest Central
ProQuest One Applied & Life Sciences
ProQuest Engineering Collection
ProQuest One Academic UKI Edition
ProQuest Central Korea
Materials Science & Engineering Collection
ProQuest Central (New)
ProQuest One Academic
ProQuest One Academic (New)
Engineering Collection
DatabaseTitleList Publicly Available Content Database
Database_xml – sequence: 1
  dbid: PIMPY
  name: Publicly Available Content Database
  url: http://search.proquest.com/publiccontent
  sourceTypes: Aggregation Database
DeliveryMethod fulltext_linktorsrc
Discipline Physics
EISSN 2331-8422
Genre Working Paper/Pre-Print
GroupedDBID 8FE
8FG
ABJCF
ABUWG
AFKRA
ALMA_UNASSIGNED_HOLDINGS
AZQEC
BENPR
BGLVJ
CCPQU
DWQXO
FRJ
HCIFZ
L6V
M7S
M~E
PHGZM
PHGZT
PIMPY
PKEHL
PQEST
PQGLB
PQQKQ
PQUKI
PRINS
PTHSS
ID FETCH-LOGICAL-a529-32b5c6e7e091e3a29345e98db8dc81cae91522a35ba227b32d69a096c557e8dc3
IEDL.DBID M7S
IngestDate Mon Jun 30 09:23:16 EDT 2025
IsDoiOpenAccess true
IsOpenAccess true
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a529-32b5c6e7e091e3a29345e98db8dc81cae91522a35ba227b32d69a096c557e8dc3
Notes SourceType-Working Papers-1
ObjectType-Working Paper/Pre-Print-1
content type line 50
OpenAccessLink https://www.proquest.com/docview/2221106519?pq-origsite=%requestingapplication%
PQID 2221106519
PQPubID 2050157
ParticipantIDs proquest_journals_2221106519
PublicationCentury 2000
PublicationDate 20190518
PublicationDateYYYYMMDD 2019-05-18
PublicationDate_xml – month: 05
  year: 2019
  text: 20190518
  day: 18
PublicationDecade 2010
PublicationPlace Ithaca
PublicationPlace_xml – name: Ithaca
PublicationTitle arXiv.org
PublicationYear 2019
Publisher Cornell University Library, arXiv.org
Publisher_xml – name: Cornell University Library, arXiv.org
SSID ssj0002672553
Score 1.6883845
SecondaryResourceType preprint
Snippet High-level programming languages such as Python are increasingly used to provide intuitive interfaces to libraries written in lower-level languages and for...
SourceID proquest
SourceType Aggregation Database
SubjectTerms Cosmology
Data management
Dependence
Distance learning
Fault tolerance
Machine learning
Materials science
Migration
Moore's law
Parallel programming
Programming languages
Provisioning
Python
Title Parsl: Pervasive Parallel Programming in Python
URI https://www.proquest.com/docview/2221106519
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV09T8MwELWgBYmJb_FRqgysbhs7TmwWJFArGKgs6FCmyh-HVKm0JSkV_HvOaQoDEgtjnAyxbN17d_d0j5BL5OjcIHGlsXGGJiJ2VKUKqABAAPSYFklbmk1k_b4cDpWuCm5FJatcx8QyUPuZCzXyNuIYIlWKhON6_kaDa1TorlYWGpukHqYkxKV07-m7xsLSDBkzXzUzy9FdbZN_jJctREHRCmgnf4XgEld6u__9oz1S12YO-T7ZgOkB2S71nK44JG2NCevkKtKh5hoU6hEuBNuUSaRXiqxXxKxoPI30Z5gecEQGve7g9o5W3gjUCKYoZ1a4FDJAuAduELMTAUp6K72TsTOgEJeZ4cIaxjLLmU-VwWzFCZEBfsOPSW06m8IJiVzmPZPSWgCXOM5tIr0wUoSWY-fFdE5JY739UXW_i9HP3s_-fn1OdpBiqNBvj2WD1Bb5O1yQLbdcjIu8Seo33b5-bJbHhk_6_kE_fwGNaaGd
linkProvider ProQuest
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V09T8MwED2VFgQT3-KjQAYY0zZ2nNhIiAGoWrVUkejQLXIcI1UqaUlKoT-K_8g5bWBAYuvAmkRWdLbvvbt79gFcIkenEomr7UglbZc5yhae0DbTGgEwxrCIR3mzCb_X44OBCErwWZyFMbLKwifmjjoeK5MjryOOIVJ5SDhuJ6-26RplqqtFC43Fsujo-TuGbNlN-x7n94qQ5kP_rmUvuwrYkhFhUxIx5WlfI1BqKhHtXKYFjyMeK-4oqQUiGpGURZIQP6Ik9oREnq8Y8zV-Q3HYNaggiyAiVwo-fad0iOcjQaeL2ml-U1hdph_DWQ1Bl9UMuPJfHj-Hseb2PzPADlQCOdHpLpR0sgcbuVpVZftQDzAcH11bgckoG_29hQ9MU5iRFSz0Zi-IyNYwsYK5uRvhAPqr-MVDKCfjRB-Bpfw4JpxHkdbKVZRGLo-Z5MwUVBvPsnEM1cLa4XL3ZuGPqU_-fn0Bm63-YzfstnudU9hCMiWMssDhVShP0zd9ButqNh1m6Xm-UiwIVzwxX3CK-ac
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Parsl%3A+Pervasive+Parallel+Programming+in+Python&rft.jtitle=arXiv.org&rft.au=Babuji%2C+Yadu&rft.au=Woodard%2C+Anna&rft.au=Li%2C+Zhuozhao&rft.au=Katz%2C+Daniel+S&rft.date=2019-05-18&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422&rft_id=info:doi/10.48550%2Farxiv.1905.02158