Dynamic DAG scheduling for many-task computing of distributed eco-hydrological model

The computing of distributed hydrological model at large scale is increasingly characterized by data intensive and computation intensive, especially for the multi-process coupling model. Parallel computing is one effective approach to cope with this situation. The easily extensible fine-grained para...

Full description

Saved in:
Bibliographic Details
Published in:The Journal of supercomputing Vol. 75; no. 2; pp. 510 - 532
Main Authors: Yue, Shasha, Ma, Yan, Chen, Lajiao, Wang, Yuzhu, Song, Weijing
Format: Journal Article
Language:English
Published: New York Springer US 06.02.2019
Springer Nature B.V
Subjects:
Compilers
Computer Science
Computing time
Critical path
Hydrologic models
Hydrology
Interpreters
Parallel processing
Processor Architectures
Programming Languages
Schedules
Task scheduling
Parallel computing
Many-task computing
Hydrological model
DAG scheduling
ISSN:0920-8542, 1573-0484
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The computing of distributed hydrological model at large scale is increasingly characterized by data intensive and computation intensive, especially for the multi-process coupling model. Parallel computing is one effective approach to cope with this situation. The easily extensible fine-grained parallelization method can substantially improve the computing efficiency. Based on many-task computing, we proposed a parallel scheme that the whole computing of the distributed hydrological model is split into tremendous amount of small sub-tasks which are directly dispatched into the cluster nodes by the traditional local resource managers (LRMs). The task-splitting method, the single task model and the representation of dependencies between tasks are also proposed. In order to efficiently schedule so many tasks, a dynamic DAG scheduling method based on critical path and depth is provided. The management of intermediate file, the control strategy for LRMs and the fault recovery is also introduced to deal with the problems encountered in the actual parallel implementation process. The parallel scheme is tested with an optimality-based distributed eco-hydrological model (disVOM) in the Poyang Lake sub-basin. It is demonstrated that our approach provide efficient computing performance.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0920-8542
1573-0484
DOI:10.1007/s11227-017-2047-1