Parallel astronomical data processing with Python: Recipes for multicore machines

High performance computing has been used in various fields of astrophysical research. But most of it is implemented on massively parallel systems (supercomputers) or graphical processing unit clusters. With the advent of multicore processors in the last decade, many serial software codes have been r...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Astronomy and computing Ročník 2; s. 1 - 10
Hlavní autoři: Singh, Navtej, Browne, Lisa-Marie, Butler, Ray
Médium: Journal Article
Jazyk:angličtina
Vydáno: Elsevier B.V 01.08.2013
Témata:
Astronomical data processing
Deconvolution
Multicore programming
Parallel computing
Parallel Python
Python multiprocessing
Parallel computing
Deconvolution
Astronomical data processing
Python multiprocessing
Multicore programming
Parallel Python
ISSN:2213-1337, 2213-1345
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:High performance computing has been used in various fields of astrophysical research. But most of it is implemented on massively parallel systems (supercomputers) or graphical processing unit clusters. With the advent of multicore processors in the last decade, many serial software codes have been re-implemented in parallel mode to utilize the full potential of these processors. In this paper, we propose parallel processing recipes for multicore machines for astronomical data processing. The target audience is astronomers who use Python as their preferred scripting language and who may be using PyRAF/IRAF for data processing. Three problems of varied complexity were benchmarked on three different types of multicore processors to demonstrate the benefits, in terms of execution time, of parallelizing data processing tasks. The native multiprocessing module available in Python makes it a relatively trivial task to implement the parallel code. We have also compared the three multiprocessing approaches—Pool/Map, Process/Queue and Parallel Python. Our test codes are freely available and can be downloaded from our website. •We propose three recipes for parallelizing long running tasks on multicore machines.•Native Python multiprocessing module makes it trivial to write parallel code.•Parallel performance can be optimized by carefully load balancing the workload.•The cross-platform nature of Python makes the code portable on multiple platforms.
ISSN:2213-1337
2213-1345
DOI:10.1016/j.ascom.2013.04.002