Optimization of fault tolerance for iterative graph algorithm in spark GraphX based on high performance computing cluster

GraphX is a graph computing library based on Spark systems, where fault tolerance is a necessary guarantee for the high availability. However, the existing fault tolerance methods are mostly implemented in a pessimistic way and are aimed at general computing tasks. Considering the characteristics of...

Full description

Saved in:
Bibliographic Details
Published in:CCF transactions on high performance computing (Online) Vol. 7; no. 5; pp. 465 - 477
Main Authors: He, Mengsi, Fu, Zhongming, Tian, Wenlong
Format: Journal Article
Language:English
Published: Beijing Springer Nature B.V 01.10.2025
Subjects:
Algorithms
Clusters
Computation
Data integrity
Distributed processing
Failure
Fault tolerance
High performance computing
Iterative methods
Performance evaluation
ISSN:2524-4922, 2524-4930
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:GraphX is a graph computing library based on Spark systems, where fault tolerance is a necessary guarantee for the high availability. However, the existing fault tolerance methods are mostly implemented in a pessimistic way and are aimed at general computing tasks. Considering the characteristics of iterative computation, this paper presents a combination method of the optimistic fault tolerance and checkpoint for recovering the data under different failure conditions. Firstly, for single node failure, we propose the optimistic fault tolerance mechanism based on compensation function. It does not add fault tolerance measures in advance and will not incur additional costs when there are no failures. Secondly, for multiple node failures, we propose the automatic checkpoint management strategy based on RDD importance. It comprehensively considers the factors of lineage length of RDD, dependency relationship, and computation time of RDD, which can set the RDD as the checkpoint properly. Finally, we implement our proposals in GraphX of Spark-3.5.1, and evaluate the performance by using representative iterative graph algorithms on the high performance computing cluster. The results verify the correctness of iteration results of the mechanism, and illustrate that when recovering the RDD partition, the job execution time can be reduced by the mechanism and strategy substantially.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2524-4922
2524-4930
DOI:10.1007/s42514-025-00225-2