Improved Hungarian algorithm-based task scheduling optimization strategy for remote sensing big data processing

With the development of remote sensing technology and computing science, remote sensing data present typical big data characteristics. The rapid development of remote sensing big data has brought a large number of data processing tasks, which bring huge challenges to computing. Distributed computing...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Geo-spatial information science Jg. 27; H. 4; S. 1141 - 1154
Hauptverfasser: Zhang, Sheng, Xue, Yong, Zhang, Heng, Zhou, Xiran, Li, Kaiyuan, Liu, Runze
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Wuhan Taylor & Francis 03.07.2024
Taylor & Francis Ltd
School of Computing and Engineering,University of Derby,Derby,UK
Artificial Intelligence Research Institute,China University of Mining and Technolog,Xuzhou,China
School of Environment and Spatial Informatics,China University of Mining and Technology,Xuzhou,China%School of Environment and Spatial Informatics,China University of Mining and Technology,Xuzhou,China
Artificial Intelligence Research Institute,China University of Mining and Technolog,Xuzhou,China%Shandong Province Institute of Land Surveying and Mapping,Jinan,China%School of Environment and Spatial Informatics,China University of Mining and Technology,Xuzhou,China
Taylor & Francis Group
Schlagworte:
Algorithms
Batch processing
Big Data
Data processing
Distributed processing
Hungarian algorithm
large-scale task
Optical analysis
Optical data processing
Optical thickness
optimal assignment
Optimization
Remote sensing
remote sensing big data
Resource scheduling
Scheduling
Task scheduling
Workflow
large-scale task
Workflow
optimal assignment
Hungarian algorithm
remote sensing big data
ISSN:1009-5020, 1993-5153
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:With the development of remote sensing technology and computing science, remote sensing data present typical big data characteristics. The rapid development of remote sensing big data has brought a large number of data processing tasks, which bring huge challenges to computing. Distributed computing is the primary means to process remote sensing big data, and task scheduling plays a key role in this process. This study analyzes the characteristics of batch processing of remote sensing big data. This paper uses the Hungarian algorithm as a basis for proposing a novel strategy for task assignment optimization of remote sensing big data batch workflow, called optimal sequence dynamic assignment algorithm, which is applicable to heterogeneously distributed computing environments. This strategy has two core contents: the improved Hungarian algorithm model and the multi-level optimal assignment task queue mechanism. Moreover, the strategy solves the dependency, mismatch, and computational resource idleness problems in the optimal scheduling of remote sensing batch processing tasks. The proposed strategy likewise effectively improves data processing efficiency without increasing computer hardware resources and without optimizing the computational algorithm. We experimented with the aerosol optical depth retrieval algorithm workflow using this strategy. Compared with the processing before optimization, the makespan of the proposed method was shortened by at least 20%. Compared with popular scheduling algorithm, the proposed method has evident competitiveness in acceleration effect and large-scale task scheduling.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1009-5020
1993-5153
DOI:10.1080/10095020.2023.2178339