Optimization of fault tolerance for iterative graph algorithm in spark GraphX based on high performance computing cluster

GraphX is a graph computing library based on Spark systems, where fault tolerance is a necessary guarantee for the high availability. However, the existing fault tolerance methods are mostly implemented in a pessimistic way and are aimed at general computing tasks. Considering the characteristics of...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:CCF transactions on high performance computing (Online) Ročník 7; číslo 5; s. 465 - 477
Hlavní autoři: He, Mengsi, Fu, Zhongming, Tian, Wenlong
Médium: Journal Article
Jazyk:angličtina
Vydáno: Beijing Springer Nature B.V 01.10.2025
Témata:
Algorithms
Clusters
Computation
Data integrity
Distributed processing
Failure
Fault tolerance
High performance computing
Iterative methods
Performance evaluation
ISSN:2524-4922, 2524-4930
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:GraphX is a graph computing library based on Spark systems, where fault tolerance is a necessary guarantee for the high availability. However, the existing fault tolerance methods are mostly implemented in a pessimistic way and are aimed at general computing tasks. Considering the characteristics of iterative computation, this paper presents a combination method of the optimistic fault tolerance and checkpoint for recovering the data under different failure conditions. Firstly, for single node failure, we propose the optimistic fault tolerance mechanism based on compensation function. It does not add fault tolerance measures in advance and will not incur additional costs when there are no failures. Secondly, for multiple node failures, we propose the automatic checkpoint management strategy based on RDD importance. It comprehensively considers the factors of lineage length of RDD, dependency relationship, and computation time of RDD, which can set the RDD as the checkpoint properly. Finally, we implement our proposals in GraphX of Spark-3.5.1, and evaluate the performance by using representative iterative graph algorithms on the high performance computing cluster. The results verify the correctness of iteration results of the mechanism, and illustrate that when recovering the RDD partition, the job execution time can be reduced by the mechanism and strategy substantially.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2524-4922
2524-4930
DOI:10.1007/s42514-025-00225-2