Profiling and optimization of Python-based social sciences applications on HPC systems by means of task and data parallelism

The article presents optimization techniques for two Python-based large-scale social sciences applications: SN (Social Network) Simulator and KPM (Kernel Polynomial Method). These applications use MPI technology to transfer data between computing processes, which in the regular implementation leads...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Future generation computer systems Ročník 148; s. 623 - 635
Hlavní autoři: Szustak, Lukasz, Lawenda, Marcin, Arming, Sebastian, Bankhamer, Gregor, Schweimer, Christoph, Elsässer, Robert
Médium: Journal Article
Jazyk:angličtina
Vydáno: Elsevier B.V 01.11.2023
Témata:
CcNUMA
Co-design
Data parallelism
HPC
Optimization
Profiling
Social sciences applications
Task parallelism
Data parallelism
Co-design
Profiling
Task parallelism
Social sciences applications
CcNUMA
HPC
Optimization
ISSN:0167-739X, 1872-7115
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:The article presents optimization techniques for two Python-based large-scale social sciences applications: SN (Social Network) Simulator and KPM (Kernel Polynomial Method). These applications use MPI technology to transfer data between computing processes, which in the regular implementation leads to load imbalance and performance degradation. To avoid this effect, we propose a 2-stage optimization. In the first step, the order of tasks is changed, and in the second step, the tasks are divided into smaller ones for easier allocation. In addition, we focus on mitigating performance and memory bottlenecks using modern ccNUMA systems with multiple NUMA domains. As part of the performance analysis, the limitations of communication in data traffic between and within the processor were revealed and resolved through appropriate data allocation. Benchmarking was carried out, examining various environments, including vendors of traditional x86-64 and ARM-based processors for HPC. •Profiling and optimization of python-based parallel applications.•ccNUMA aware computing.•Social sciences applications.•Load balancing for task and data parallelisms.
ISSN:0167-739X
1872-7115
DOI:10.1016/j.future.2023.07.005