Parallel astronomical data processing with Python: Recipes for multicore machines

High performance computing has been used in various fields of astrophysical research. But most of it is implemented on massively parallel systems (supercomputers) or graphical processing unit clusters. With the advent of multicore processors in the last decade, many serial software codes have been r...

Full description

Saved in:
Bibliographic Details
Published in:Astronomy and computing Vol. 2; pp. 1 - 10
Main Authors: Singh, Navtej, Browne, Lisa-Marie, Butler, Ray
Format: Journal Article
Language:English
Published: Elsevier B.V 01.08.2013
Subjects:
Astronomical data processing
Deconvolution
Multicore programming
Parallel computing
Parallel Python
Python multiprocessing
Parallel computing
Deconvolution
Astronomical data processing
Python multiprocessing
Multicore programming
Parallel Python
ISSN:2213-1337, 2213-1345
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:High performance computing has been used in various fields of astrophysical research. But most of it is implemented on massively parallel systems (supercomputers) or graphical processing unit clusters. With the advent of multicore processors in the last decade, many serial software codes have been re-implemented in parallel mode to utilize the full potential of these processors. In this paper, we propose parallel processing recipes for multicore machines for astronomical data processing. The target audience is astronomers who use Python as their preferred scripting language and who may be using PyRAF/IRAF for data processing. Three problems of varied complexity were benchmarked on three different types of multicore processors to demonstrate the benefits, in terms of execution time, of parallelizing data processing tasks. The native multiprocessing module available in Python makes it a relatively trivial task to implement the parallel code. We have also compared the three multiprocessing approaches—Pool/Map, Process/Queue and Parallel Python. Our test codes are freely available and can be downloaded from our website. •We propose three recipes for parallelizing long running tasks on multicore machines.•Native Python multiprocessing module makes it trivial to write parallel code.•Parallel performance can be optimized by carefully load balancing the workload.•The cross-platform nature of Python makes the code portable on multiple platforms.
ISSN:2213-1337
2213-1345
DOI:10.1016/j.ascom.2013.04.002