A Deep Dive into Task-Based Parallelism in Python

Modern Python programs in high-performance computing call into compiled libraries and kernels for performance-critical tasks. However, effectively parallelizing these finer-grained, and often dynamic, kernels across modern heterogeneous platforms remains a challenge. First, we perform an experimenta...

Full description

Saved in:
Bibliographic Details
Published in:2024 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) pp. 1147 - 1149
Main Authors: Ruys, William, Lee, Hochan, You, Bozhi, Talati, Shreya, Park, Jaeyoung, Almgren-Bell, James, Yan, Yineng, Fernando, Milinda, Biros, George, Erez, Mattan, Burtscher, Martin, Rossbach, Christopher J., Pingali, Keshav, Gligoric, Milos
Format: Conference Proceeding
Language:English
Published: IEEE 27.05.2024
Subjects:
heterogeneous computing
High performance computing
Libraries
Parallel processing
parallel programming
Performance evaluation
Processor scheduling
Programming
Python
Runtime
task runtime
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Modern Python programs in high-performance computing call into compiled libraries and kernels for performance-critical tasks. However, effectively parallelizing these finer-grained, and often dynamic, kernels across modern heterogeneous platforms remains a challenge. First, we perform an experimental study to examine the impact of Python's Global Interpreter Lock (GIL), and potential speedups under a GIL-less PEP703 future, to guide runtime design. Using our optimized runtime, we explore scheduling tasks with constraints that require resources across multiple, potentially diverse, devices through the introduction of new programming abstractions and runtime mechanisms. We extend an existing Python tasking library, Parla, to augment its performance and add support for such multi-device tasks. Our experimental analysis, using tasks graphs from synthetic and real applications, shows at least a 3 ×(and up to 6 ×) performance improvement over its predecessor in scenarios with high GIL contention. When scheduling multi-GPU tasks, we observe an 8x reduction in per-task launching overhead compared to a multi-process system.
DOI:10.1109/IPDPSW63119.2024.00187