Enabling Low-Overhead HT-HPC Workflows at Extreme Scale using GNU Parallel

GNU Parallel is a versatile and powerful tool for process parallelization widely used in scientific computing. This paper demonstrates its effective application in high-performance computing (HPC) environments, particularly focusing on its scalability and efficiency in executing large-scale high-thr...

Full description

Saved in:
Bibliographic Details
Published in:SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis pp. 2056 - 2063
Main Authors: Maheshwari, Ketan, Arndt, William, Karimi, Ahmad Maroof, Yin, Junqi, Suter, Frederic, Johnson, Seth, Da Silva, Rafael Ferreira
Format: Conference Proceeding
Language:English
Published: IEEE 17.11.2024
Subjects:
Data transfer
Distributed Computing
GNU Parallel
Graphics processing units
High performance computing
High-Throughput Computing
Python
Resource management
Scalability
Scientific computing
Scientific Workflows
Supercomputers
Transforms
Workflow management software
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:GNU Parallel is a versatile and powerful tool for process parallelization widely used in scientific computing. This paper demonstrates its effective application in high-performance computing (HPC) environments, particularly focusing on its scalability and efficiency in executing large-scale high-throughput high-performance computing (HT-HPC) workflows. Through real-world examples, we highlight GNU Parallel's performance across various HPC workloads, including GPU computing, container-based workloads, and node-local NVMe storage. Our results on two leading supercomputers, OLCF's Frontier and NERSC's Perlmutter, showcase GNU Parallel's rapid process dispatching ability and its capacity to maintain low overhead even at extreme scales. We explore GNU Parallel's application in massive parallel file transfers using a scheduled Data Transfer Node (DTN) cluster, emphasizing its broad utility in diverse scientific workflows. Beyond its direct application as a viable workflow manager, GNU Parallel can be employed in conjunction with other workflow systems as a "last-mile" parallelizing driver and as a quick prototyping tool to design and extract parallel profiles from application executions. We then argue that the potential for GNU Parallel to transform workflow management at extreme scales is substantial, paving the way for more efficient and effective scientific discoveries.
DOI:10.1109/SCW63240.2024.00257