PHDFS: Optimizing I/O performance of HDFS in deep learning cloud computing platform

For deep learning cloud computing platforms, file system is a fundamental and critical component. Hadoop distributed file system (HDFS) is widely used in large scale clusters due to its high performance and high availability. However, in deep learning datasets, the number of files is huge but the fi...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Journal of systems architecture Ročník 109; s. 101810
Hlavní autori: Zhu, Zongwei, Tan, Luchao, Li, Yinzhen, Ji, Cheng
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Elsevier B.V 01.10.2020
Predmet:
Cloud computing
Deep learning
Distributed file system
Small files
Deep learning
Cloud computing
Small files
Distributed file system
ISSN:1383-7621, 1873-6165
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:For deep learning cloud computing platforms, file system is a fundamental and critical component. Hadoop distributed file system (HDFS) is widely used in large scale clusters due to its high performance and high availability. However, in deep learning datasets, the number of files is huge but the file size is small, making HDFS suffer a severe performance penalty. Although there have been many optimizing methods for addressing the small file problem, none of them take the file correlation in deep learning datasets into consideration. To address such problem, this paper proposes a Pile-HDFS (PHDFS) based on a new file aggregation approach. Pile is designed as the I/O unit merging a group of small files according to their correlation. In order to effectively access small files, we design a two-layer manager and add the inner organization information to data blocks. Experimental results demonstrate that, compared with the original HDFS, PHDFS can dramatically decrease the latency when accessing small files and improve the FPS (Frames Per Second) of typical deep learning models by 40%.
ISSN:1383-7621
1873-6165
DOI:10.1016/j.sysarc.2020.101810