A multilevel algorithm for scalable independent task assignment

Assigning a large number of independent tasks to heterogeneous processors is a fundamental problem in modern computing, with applications in many domains such as cloud services, web crawling, and AI training. Exact and matheuristic approaches deliver high-quality assignments but incur superlinear or...

Full description

Saved in:
Bibliographic Details
Published in:Future generation computer systems Vol. 176; p. 108183
Main Authors: Burhan Tabak, H., Kartal Tabak, E., Aykanat, Cevdet
Format: Journal Article
Language:English
Published: Elsevier B.V 01.03.2026
Subjects:
Distributed and parallelized LLM training
Geographically distributed web crawling
Heterogeneous systems
Independent task assignment
Load balancing
Multilevel framework
Parallel and distributed computing
Heterogeneous systems
Multilevel framework
Parallel and distributed computing
Independent task assignment
Geographically distributed web crawling
Distributed and parallelized LLM training
Load balancing
ISSN:0167-739X
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Assigning a large number of independent tasks to heterogeneous processors is a fundamental problem in modern computing, with applications in many domains such as cloud services, web crawling, and AI training. Exact and matheuristic approaches deliver high-quality assignments but incur superlinear or even exponential runtime costs, making them impractical, especially on large problem instances. Conversely, lightweight heuristics run efficiently at scale but often produce assignments with much lower quality. To address this issue, we present the first multilevel framework for the independent task assignment problem that maintains an end-to-end linear runtime bound of O(KN), where K×N is the size of the expected-time-to-compute matrix, with K and N respectively representing the number of processors and tasks. We propose (i) novel high-quality coarsening metrics that numerically define task characteristics and similarity; (ii) an efficient and effective matching algorithm that incorporates these metrics while maintaining linear time complexity with respect to the input size; (iii) an initial solution scheme that generates base solutions using complementary heuristics, which are disjointly projected back through the uncoarsening levels; (iv) an effective and efficient uncoarsening algorithm that iteratively improves assignment quality with different refinement algorithms. Extensive experimental evaluations involving hundreds of millions of tasks demonstrate that our algorithm achieves significantly higher quality and runs faster than known high-quality heuristics, making it a practical choice for the problem instances at high scale.
ISSN:0167-739X
DOI:10.1016/j.future.2025.108183