Python Workflows on HPC Systems

The recent successes and wide spread application of compute intensive machine learning and data analytics methods have been boosting the usage of the Python programming language on HPC systems. While Python provides many advantages for the users, it has not been designed with a focus on multi-user e...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:2020 IEEE/ACM 9th Workshop on Python for High-Performance and Scientific Computing (PyHPC) s. 32 - 40
Hlavní autoři: Strasel, Dominik, Reusch, Philipp, Keuper, Janis
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 01.11.2020
Témata:
compute intensive machine learning
data analysis
data analytics
data analytics methods
data structures
deep learning
Deep Learning applications
Graphics processing units
high performance computing
HPC clusters
HPC system
Kernel
learning (artificial intelligence)
Linux
Machine learning
multiuser environments
multiuser Python software environments
parallel processing
parallel programming
Python
Python jobs
Python programming language
recent successes
secure Python workflows
Software
software performance evaluation
stable Python workflows
wide spread application
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:The recent successes and wide spread application of compute intensive machine learning and data analytics methods have been boosting the usage of the Python programming language on HPC systems. While Python provides many advantages for the users, it has not been designed with a focus on multi-user environments or parallel programming - making it quite challenging to maintain stable and secure Python workflows on a HPC system. In this paper, we analyze the key problems induced by the usage of Python on HPC clusters and sketch appropriate workarounds for efficiently maintaining multi-user Python software environments, securing and restricting resources of Python jobs and containing Python processes, while focusing on Deep Learning applications running on GPU clusters.
DOI:10.1109/PyHPC51966.2020.00009