Parsl+CWL: Towards Combining the Python and CWL Ecosystems

The Common Workflow Language (CWL) is a widely adopted language for defining and sharing computational workflows. It is designed to be independent of the execution engine on which workflows are executed. In this paper, we describe our experiences integrating CWL with Parsl, a Python-based parallel p...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis s. 2040 - 2047
Hlavní autoři: Karle, Nishchay, Clifford, Ben, Babuji, Yadu, Chard, Ryan, Katz, Daniel S., Chard, Kyle
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 17.11.2024
Témata:
commandlinetool
common workflow language
common workflow language (cwl)
computational reproducibility
Conferences
cross-platform execution
cwl
dataflow programming
dynamic workflows
Ecosystems
Engines
High performance computing
high performance computing (hpc)
Image processing
inline expressions
Libraries
parallel computing
Parallel programming
parsl
Python
python programming
resource management
scalability
scientific workflows
workflow
workflow automation
workflow ecosystem
workflow execution
workflow integration
workflow management systems
workflow portability
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:The Common Workflow Language (CWL) is a widely adopted language for defining and sharing computational workflows. It is designed to be independent of the execution engine on which workflows are executed. In this paper, we describe our experiences integrating CWL with Parsl, a Python-based parallel programming library designed to manage execution of workflows across diverse computing environments. We propose a new method that converts CWL CommandLineTool definitions into Parsl apps, enabling Parsl scripts to easily import and use tools represented in CWL. We describe a Parsl runner that is capable of executing a CWL CommandLineTool directly. We also describe a proof-of-concept extension to support inline Python in a CWL workflow definition, enabling seamless use in Parsl's Python ecosystem. We demonstrate the benefits of this integration by presenting example CWL CommandLineTool definitions that show how they can be used in Parsl, and comparing performance of executing an image processing workflow using the Parsl integration and other CWL runners.
DOI:10.1109/SCW63240.2024.00255