PHDFS: Optimizing I/O performance of HDFS in deep learning cloud computing platform

For deep learning cloud computing platforms, file system is a fundamental and critical component. Hadoop distributed file system (HDFS) is widely used in large scale clusters due to its high performance and high availability. However, in deep learning datasets, the number of files is huge but the fi...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Journal of systems architecture Ročník 109; s. 101810
Hlavní autoři: Zhu, Zongwei, Tan, Luchao, Li, Yinzhen, Ji, Cheng
Médium: Journal Article
Jazyk:angličtina
Vydáno: Elsevier B.V 01.10.2020
Témata:
Cloud computing
Deep learning
Distributed file system
Small files
Deep learning
Cloud computing
Small files
Distributed file system
ISSN:1383-7621, 1873-6165
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:For deep learning cloud computing platforms, file system is a fundamental and critical component. Hadoop distributed file system (HDFS) is widely used in large scale clusters due to its high performance and high availability. However, in deep learning datasets, the number of files is huge but the file size is small, making HDFS suffer a severe performance penalty. Although there have been many optimizing methods for addressing the small file problem, none of them take the file correlation in deep learning datasets into consideration. To address such problem, this paper proposes a Pile-HDFS (PHDFS) based on a new file aggregation approach. Pile is designed as the I/O unit merging a group of small files according to their correlation. In order to effectively access small files, we design a two-layer manager and add the inner organization information to data blocks. Experimental results demonstrate that, compared with the original HDFS, PHDFS can dramatically decrease the latency when accessing small files and improve the FPS (Frames Per Second) of typical deep learning models by 40%.
ISSN:1383-7621
1873-6165
DOI:10.1016/j.sysarc.2020.101810