Byzantine Resilient Non-Convex SCSG With Distributed Batch Gradient Computations

Distributed learning is an important paradigm in the current machine learning algorithms with large datasets. In this paper, distributed stochastic optimization problem of minimizing a nonconvex function in an adversarial setting is considered. A robust variant of the stochastic variance-reduced alg...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on signal and information processing over networks Vol. 7; pp. 754 - 766
Main Authors: Bulusu, Saikiran, Khanduri, Prashant, Kafle, Swatantra, Sharma, Pranay, Varshney, Pramod K.
Format: Journal Article
Language:English
Published: Piscataway IEEE 2021
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Algorithms
Computation
Convergence
Datasets
distributed optimization
Machine learning
Machine learning algorithms
Nonconvex optimization
Optimization
Performance evaluation
Stochastic gradient descent
Stochastic processes
variance reduction
ISSN:2373-776X, 2373-7778
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Distributed learning is an important paradigm in the current machine learning algorithms with large datasets. In this paper, distributed stochastic optimization problem of minimizing a nonconvex function in an adversarial setting is considered. A robust variant of the stochastic variance-reduced algorithm is proposed. In the distributed setup, we assume that a fraction of worker nodes (WNs) can be Byzantines. We assume that the batch gradients are computed at the WNs and the stochastic gradients are computed at the central node (CN). We provide the convergence rate of the proposed algorithm which employs the design of a novel filtering rule that is independent of the problem dimension. Furthermore, we capture the effect of Byzantines present in the network on the convergence performance of the algorithm. We evaluate the performance of the proposed algorithm and present the simulation results using real world datasets, in addition to providing the theoretical guarantees.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2373-776X
2373-7778
DOI:10.1109/TSIPN.2021.3129352