Distributed Gradient Descent Algorithm Robust to an Arbitrary Number of Byzantine Attackers

Due to the growth of modern dataset size and the desire to harness computing power of multiple machines, there is a recent surge of interest in the design of distributed machine learning algorithms. However, distributed algorithms are sensitive to Byzantine attackers who can send falsified data to p...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on signal processing Vol. 67; no. 22; pp. 5850 - 5864
Main Authors: Cao, Xinyang, Lai, Lifeng
Format: Journal Article
Language:English
Published: New York IEEE 15.11.2019
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Algorithms
Byzantine attacker
Convergence
Datasets
distributed gradient descent
Ground truth
Machine learning
Optimization
Radio frequency
Robustness (mathematics)
Servers
Signal processing algorithms
Sociology
Statistics
ISSN:1053-587X, 1941-0476
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Due to the growth of modern dataset size and the desire to harness computing power of multiple machines, there is a recent surge of interest in the design of distributed machine learning algorithms. However, distributed algorithms are sensitive to Byzantine attackers who can send falsified data to prevent the convergence of algorithms or lead the algorithms to converge to value of the attackers' choice. Some recent work proposed interesting algorithms that can deal with the scenario when up to half of the workers are compromised. In this paper, we propose a novel algorithm that can deal with an arbitrary number of Byzantine attackers. The main idea is to ask the parameter server to randomly select a small clean dataset and compute noisy gradient using this small dataset. This noisy gradient will then be used as a ground truth to filter out information sent by compromised workers. We show that the proposed algorithm converges to the neighborhood of the population minimizer regardless the number of Byzantine attackers. We further provide numerical examples to show that the proposed algorithm can benefit from the presence of good workers and achieve better performance than existing algorithms.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1053-587X
1941-0476
DOI:10.1109/TSP.2019.2946020