PHDFS: Optimizing I/O performance of HDFS in deep learning cloud computing platform

For deep learning cloud computing platforms, file system is a fundamental and critical component. Hadoop distributed file system (HDFS) is widely used in large scale clusters due to its high performance and high availability. However, in deep learning datasets, the number of files is huge but the fi...

Full description

Saved in:
Bibliographic Details
Published in:Journal of systems architecture Vol. 109; p. 101810
Main Authors: Zhu, Zongwei, Tan, Luchao, Li, Yinzhen, Ji, Cheng
Format: Journal Article
Language:English
Published: Elsevier B.V 01.10.2020
Subjects:
Cloud computing
Deep learning
Distributed file system
Small files
Deep learning
Cloud computing
Small files
Distributed file system
ISSN:1383-7621, 1873-6165
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:For deep learning cloud computing platforms, file system is a fundamental and critical component. Hadoop distributed file system (HDFS) is widely used in large scale clusters due to its high performance and high availability. However, in deep learning datasets, the number of files is huge but the file size is small, making HDFS suffer a severe performance penalty. Although there have been many optimizing methods for addressing the small file problem, none of them take the file correlation in deep learning datasets into consideration. To address such problem, this paper proposes a Pile-HDFS (PHDFS) based on a new file aggregation approach. Pile is designed as the I/O unit merging a group of small files according to their correlation. In order to effectively access small files, we design a two-layer manager and add the inner organization information to data blocks. Experimental results demonstrate that, compared with the original HDFS, PHDFS can dramatically decrease the latency when accessing small files and improve the FPS (Frames Per Second) of typical deep learning models by 40%.
ISSN:1383-7621
1873-6165
DOI:10.1016/j.sysarc.2020.101810