A multilevel algorithm for scalable independent task assignment

Assigning a large number of independent tasks to heterogeneous processors is a fundamental problem in modern computing, with applications in many domains such as cloud services, web crawling, and AI training. Exact and matheuristic approaches deliver high-quality assignments but incur superlinear or...

Celý popis

Uloženo v:

Podrobná bibliografie
Vydáno v:	Future generation computer systems Ročník 176; s. 108183
Hlavní autoři:	Burhan Tabak, H., Kartal Tabak, E., Aykanat, Cevdet
Médium:	Journal Article
Jazyk:	angličtina
Vydáno:	Elsevier B.V 01.03.2026
Témata:	Distributed and parallelized LLM training Geographically distributed web crawling Heterogeneous systems Independent task assignment Load balancing Multilevel framework Parallel and distributed computing Heterogeneous systems Multilevel framework Parallel and distributed computing Independent task assignment Geographically distributed web crawling Distributed and parallelized LLM training Load balancing
ISSN:	0167-739X
On-line přístup:	Získat plný text
Tagy:	Přidat tag Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!

Popis
Shrnutí:	Assigning a large number of independent tasks to heterogeneous processors is a fundamental problem in modern computing, with applications in many domains such as cloud services, web crawling, and AI training. Exact and matheuristic approaches deliver high-quality assignments but incur superlinear or even exponential runtime costs, making them impractical, especially on large problem instances. Conversely, lightweight heuristics run efficiently at scale but often produce assignments with much lower quality. To address this issue, we present the first multilevel framework for the independent task assignment problem that maintains an end-to-end linear runtime bound of O(KN), where K×N is the size of the expected-time-to-compute matrix, with K and N respectively representing the number of processors and tasks. We propose (i) novel high-quality coarsening metrics that numerically define task characteristics and similarity; (ii) an efficient and effective matching algorithm that incorporates these metrics while maintaining linear time complexity with respect to the input size; (iii) an initial solution scheme that generates base solutions using complementary heuristics, which are disjointly projected back through the uncoarsening levels; (iv) an effective and efficient uncoarsening algorithm that iteratively improves assignment quality with different refinement algorithms. Extensive experimental evaluations involving hundreds of millions of tasks demonstrate that our algorithm achieves significantly higher quality and runs faster than known high-quality heuristics, making it a practical choice for the problem instances at high scale.
ISSN:	0167-739X
DOI:	10.1016/j.future.2025.108183