Parsl+CWL: Towards Combining the Python and CWL Ecosystems
The Common Workflow Language (CWL) is a widely adopted language for defining and sharing computational workflows. It is designed to be independent of the execution engine on which workflows are executed. In this paper, we describe our experiences integrating CWL with Parsl, a Python-based parallel p...
Gespeichert in:
| Veröffentlicht in: | SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis S. 2040 - 2047 |
|---|---|
| Hauptverfasser: | , , , , , |
| Format: | Tagungsbericht |
| Sprache: | Englisch |
| Veröffentlicht: |
IEEE
17.11.2024
|
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | The Common Workflow Language (CWL) is a widely adopted language for defining and sharing computational workflows. It is designed to be independent of the execution engine on which workflows are executed. In this paper, we describe our experiences integrating CWL with Parsl, a Python-based parallel programming library designed to manage execution of workflows across diverse computing environments. We propose a new method that converts CWL CommandLineTool definitions into Parsl apps, enabling Parsl scripts to easily import and use tools represented in CWL. We describe a Parsl runner that is capable of executing a CWL CommandLineTool directly. We also describe a proof-of-concept extension to support inline Python in a CWL workflow definition, enabling seamless use in Parsl's Python ecosystem. We demonstrate the benefits of this integration by presenting example CWL CommandLineTool definitions that show how they can be used in Parsl, and comparing performance of executing an image processing workflow using the Parsl integration and other CWL runners. |
|---|---|
| AbstractList | The Common Workflow Language (CWL) is a widely adopted language for defining and sharing computational workflows. It is designed to be independent of the execution engine on which workflows are executed. In this paper, we describe our experiences integrating CWL with Parsl, a Python-based parallel programming library designed to manage execution of workflows across diverse computing environments. We propose a new method that converts CWL CommandLineTool definitions into Parsl apps, enabling Parsl scripts to easily import and use tools represented in CWL. We describe a Parsl runner that is capable of executing a CWL CommandLineTool directly. We also describe a proof-of-concept extension to support inline Python in a CWL workflow definition, enabling seamless use in Parsl's Python ecosystem. We demonstrate the benefits of this integration by presenting example CWL CommandLineTool definitions that show how they can be used in Parsl, and comparing performance of executing an image processing workflow using the Parsl integration and other CWL runners. |
| Author | Chard, Kyle Babuji, Yadu Clifford, Ben Katz, Daniel S. Karle, Nishchay Chard, Ryan |
| Author_xml | – sequence: 1 givenname: Nishchay surname: Karle fullname: Karle, Nishchay organization: University of Chicago,Department of Computer Science,Chicago,USA – sequence: 2 givenname: Ben surname: Clifford fullname: Clifford, Ben organization: University of Chicago,Department of Computer Science,Chicago,USA – sequence: 3 givenname: Yadu surname: Babuji fullname: Babuji, Yadu organization: University of Chicago,Department of Computer Science,Chicago,USA – sequence: 4 givenname: Ryan surname: Chard fullname: Chard, Ryan organization: Argonne National Laboratory,Lemont,USA – sequence: 5 givenname: Daniel S. surname: Katz fullname: Katz, Daniel S. organization: Argonne National Laboratory,Lemont,USA – sequence: 6 givenname: Kyle surname: Chard fullname: Chard, Kyle organization: University of Chicago,Department of Computer Science,Chicago,USA |
| BookMark | eNotjM1KxDAYRSMoqGOfQBfZS-uXny9JZydl_IGCA47MckjbxClME2kK0re3onC4d3Eu95qchxgcIbcMCsagfHiv9kpwCQUHLgsAjnhGslKXRiAIRJTikmQp9Q0oQCPB4BVZb-2YTvfVvl7TXfy2Y5doFYemD334pNPR0e08HWOgNnR0WdFNG9OcJjekG3Lh7Sm57L9X5ONps6te8vrt-bV6rHPLNJtyoaQuHTLmfwHlW1AKWym19yiZL7XxrtGq6UDjkl5ogFZ3zHuzSClW5O7vt3fOHb7GfrDjfGBg-AIXPy-SRwk |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK RIE RIL |
| DOI | 10.1109/SCW63240.2024.00255 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP All) 1998-Present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| EISBN | 9798350355543 |
| EndPage | 2047 |
| ExternalDocumentID | 10820822 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: National Science Foundation funderid: 10.13039/100000001 – fundername: U.S. Department of Energy funderid: 10.13039/100000015 |
| GroupedDBID | 6IE 6IL ACM ALMA_UNASSIGNED_HOLDINGS CBEJK RIE RIL |
| ID | FETCH-LOGICAL-a171t-36479e511f11f106fc0665c447ff541f978feb76bd0756bdf3700c7d1ff8f9743 |
| IEDL.DBID | RIE |
| IngestDate | Wed Aug 27 01:59:34 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | false |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a171t-36479e511f11f106fc0665c447ff541f978feb76bd0756bdf3700c7d1ff8f9743 |
| PageCount | 8 |
| ParticipantIDs | ieee_primary_10820822 |
| PublicationCentury | 2000 |
| PublicationDate | 2024-Nov.-17 |
| PublicationDateYYYYMMDD | 2024-11-17 |
| PublicationDate_xml | – month: 11 year: 2024 text: 2024-Nov.-17 day: 17 |
| PublicationDecade | 2020 |
| PublicationTitle | SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis |
| PublicationTitleAbbrev | SC-W |
| PublicationYear | 2024 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssib060584085 |
| Score | 1.8893534 |
| Snippet | The Common Workflow Language (CWL) is a widely adopted language for defining and sharing computational workflows. It is designed to be independent of the... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 2040 |
| SubjectTerms | commandlinetool common workflow language common workflow language (cwl) computational reproducibility Conferences cross-platform execution cwl dataflow programming dynamic workflows Ecosystems Engines High performance computing high performance computing (hpc) Image processing inline expressions Libraries parallel computing Parallel programming parsl Python python programming resource management scalability scientific workflows workflow workflow automation workflow ecosystem workflow execution workflow integration workflow management systems workflow portability |
| Title | Parsl+CWL: Towards Combining the Python and CWL Ecosystems |
| URI | https://ieeexplore.ieee.org/document/10820822 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlZ3PS8MwFMeDGx48qVjxNzl4jSZZlqS7jg0PMgpOttto0zzw0sl-CP73vpd16sWDUEppCyVtw8vLy_fzZezeEeI7D16YUpfCgA2iCiBF3pc2kk-5DjKZTbjJxM_nedGK1ZMWJsaYFp_FBzpMtfx6GbY0VYY9HOMVRrQO6zhnd2Kt_c9D5T2idbVkISXzx5fhjGDkErNATYxsTXq-Xx4qKYSMj__58BOW_YjxePEdZk7ZQWzO2KDAhBTT_tnzgE_Tytc1x65dJbsHjoM6XnwSFYCXTc3xLj4Kyx20eZ2x1_FoOnwSrQ2CKJVTG0GE9zziwAhokxYClUuCMQ6gbxRgHgixcraqMfzjHnpOyuBqBeDxoumds26zbOIF4yXRYHTfS--08VFWta1BVjlo60FZe8kyavjifUe6WOzbfPXH-Wt2RO-WtHnK3bDuZrWNt-wwfGze1qu79H2-AMImjp4 |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlZ3PS8MwFMcfOgU9qTjxtzl4jSZdlqS7DsfEOQpOttto0zzw0sl-CP735mWdevEglFLaQknb8PLy8v18AW4NIb5TZ7nKk5wr1I4XDgVP20J78ilPnIhmE2Y4tJNJmtVi9aiF8d7HxWf-jg5jLb-cuRVNlYUeHuJViGjbsEPWWbVca_P7UIGPeF01W0iK9P6lOyYcuQh5YEKU7IQUfb9cVGIQ6R388_GH0PyR47HsO9AcwZavjqGThZQ0JP7jQYeN4trXBQudu4iGDywM61j2SVwAllclC3exBzdbY5sXTXjtPYy6fV4bIfBcGrnkxHhPfRgaIW1Co6OCiVPKILaVxJAJoi-MLsowAAh7bBkhnCklog0XVesEGtWs8qfAcuLBJG0rrEmU9aIodYmiSDHRFqXWZ9Ckhk_f16yL6abN53-cv4G9_uh5MB08Dp8uYJ_eMyn1pLmExnK-8lew6z6Wb4v5dfxWX2Yrkec |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=SC24-W%3A+Workshops+of+the+International+Conference+for+High+Performance+Computing%2C+Networking%2C+Storage+and+Analysis&rft.atitle=Parsl%2BCWL%3A+Towards+Combining+the+Python+and+CWL+Ecosystems&rft.au=Karle%2C+Nishchay&rft.au=Clifford%2C+Ben&rft.au=Babuji%2C+Yadu&rft.au=Chard%2C+Ryan&rft.date=2024-11-17&rft.pub=IEEE&rft.spage=2040&rft.epage=2047&rft_id=info:doi/10.1109%2FSCW63240.2024.00255&rft.externalDocID=10820822 |