Scheduling Distributed Clusters of Parallel Machines : Primal-Dual and LP-based Approximation Algorithms

The Map-Reduce computing framework rose to prominence with datasets of such size that dozens of machines on a single cluster were needed for individual jobs. As datasets approach the exabyte scale, a single job may need distributed processing not only on multiple machines, but on multiple clusters ....

Full description

Saved in:
Bibliographic Details
Published in:Algorithmica Vol. 80; no. 10; pp. 2777 - 2798
Main Authors: Murray, Riley, Khuller, Samir, Chao, Megan
Format: Journal Article
Language:English
Published: New York Springer US 01.10.2018
Springer Nature B.V
Subjects:
Algorithm Analysis and Problem Complexity
Algorithms
Approximation
Clusters
Combinatorial analysis
Completion time
Computer Science
Computer Systems Organization and Communication Networks
Data Structures and Information Theory
Datasets
Distributed processing
Employment
Mapping
Mathematics of Computing
Production scheduling
Scheduling
Theory of Computation
Approximation algorithms
Primal-dual algorithms
Machine scheduling
F.2.2 Nonnumerical Algorithms and Problems
LP relaxations
Distributed computing
ISSN:0178-4617, 1432-0541
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The Map-Reduce computing framework rose to prominence with datasets of such size that dozens of machines on a single cluster were needed for individual jobs. As datasets approach the exabyte scale, a single job may need distributed processing not only on multiple machines, but on multiple clusters . We consider a scheduling problem to minimize weighted average completion time of n jobs on m distributed clusters of parallel machines. In keeping with the scale of the problems motivating this work, we assume that (1) each job is divided into m “subjobs” and (2) distinct subjobs of a given job may be processed concurrently. When each cluster is a single machine, this is the NP-Hard concurrent open shop problem. A clear limitation of such a model is that a serial processing assumption sidesteps the issue of how different tasks of a given subjob might be processed in parallel. Our algorithms explicitly model clusters as pools of resources and effectively overcome this issue. Under a variety of parameter settings, we develop two constant factor approximation algorithms for this problem. The first algorithm uses an LP relaxation tailored to this problem from prior work. This LP-based algorithm provides strong performance guarantees. Our second algorithm exploits a surprisingly simple mapping to the special case of one machine per cluster. This mapping-based algorithm is combinatorial and extremely fast. These are the first constant factor approximations for this problem.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0178-4617
1432-0541
DOI:10.1007/s00453-017-0345-x