Enhancing Throughput of Hadoop Distributed File System for Interaction-Intensive Tasks

The performance of the Hadoop Distributed File System (HDFS)decreases dramatically when handling interaction-intensive files, i.e., files that have relatively small size but are accessed frequently. The paper analyzes the cause of throughput degradation issue when accessing interaction-intensive fil...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Proceedings - Euromicro Workshop on Parallel and Distributed Processing S. 508 - 511
Hauptverfasser: Xiayu Hua, Hao Wu, Shangping Ren
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: IEEE 01.02.2014
Schlagworte:
Arrays
Cache
HDFS
Hierarchical structure
Multi-layer neural network
Periodic structures
PSO
Resource management
Storage Allocation Algorithm
Throughput
Vectors
ISSN:1066-6192
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Abstract The performance of the Hadoop Distributed File System (HDFS)decreases dramatically when handling interaction-intensive files, i.e., files that have relatively small size but are accessed frequently. The paper analyzes the cause of throughput degradation issue when accessing interaction-intensive files and presents an enhanced HDFS architecture along with an associated storage allocation algorithm that overcomes the performance degradation problem. Experiments have shown that with the proposed architecture together with the associated storage allocation algorithm, the HDFS throughput for interaction-intensive files increase 300% in average with only a negligible performance decrease for large data set tasks.
AbstractList The performance of the Hadoop Distributed File System (HDFS)decreases dramatically when handling interaction-intensive files, i.e., files that have relatively small size but are accessed frequently. The paper analyzes the cause of throughput degradation issue when accessing interaction-intensive files and presents an enhanced HDFS architecture along with an associated storage allocation algorithm that overcomes the performance degradation problem. Experiments have shown that with the proposed architecture together with the associated storage allocation algorithm, the HDFS throughput for interaction-intensive files increase 300% in average with only a negligible performance decrease for large data set tasks.
Author Xiayu Hua
Shangping Ren
Hao Wu
Author_xml – sequence: 1
  surname: Xiayu Hua
  fullname: Xiayu Hua
  email: xhua@hawk.iit.edu
  organization: Dept. of Comput. Sci., Illinois Inst. of Technol., Chicago, IL, USA
– sequence: 2
  surname: Hao Wu
  fullname: Hao Wu
  email: hwu28@hawk.iit.edu
  organization: Dept. of Comput. Sci., Illinois Inst. of Technol., Chicago, IL, USA
– sequence: 3
  surname: Shangping Ren
  fullname: Shangping Ren
  email: ren@hawk.iit.edu
  organization: Dept. of Comput. Sci., Illinois Inst. of Technol., Chicago, IL, USA
BookMark eNotjE9PwyAcQDGZidvczZsXvkAn0BbK0eyPW7LEJVavC9AfK7pBU6jJvr0uenrvXd4EjXzwgNADJXNKiXzaL_dzRmhxrRs0k6KihZCSCSb5CI0p4TzjVLI7NInxkxAiCibH6GPlW-WN80dct30Yjm03JBws3qgmhA4vXUy900OCBq_dCfDbJSY4Yxt6vPUJemWSCz67uo_uG3Ct4le8R7dWnSLM_jlF7-tVvdhku9eX7eJ5lzkqypQpSXLQCpqKKgKWMMVlWRRUsJL_quKMCstMTjToRurCamarXBtDCRgpy3yKHv--DgAOXe_Oqr8cuKhEzlj-A6vWUwM
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/PDP.2014.110
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISBN 9781479927296
1479927295
EndPage 511
ExternalDocumentID 6787322
Genre orig-research
GroupedDBID 29N
29O
6IE
6IF
6IH
6IK
6IL
6IN
AAJGR
AAWTH
ABLEC
ACGFS
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IPLJI
M43
OCL
RIE
RIL
RNS
ID FETCH-LOGICAL-i175t-a903ebaed81a0ef02a6954417256a69a6217f2c30bebd9b4fb2f83bcc10ec9953
IEDL.DBID RIE
ISICitedReferencesCount 0
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000353964700076&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 1066-6192
IngestDate Wed Aug 27 04:50:10 EDT 2025
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i175t-a903ebaed81a0ef02a6954417256a69a6217f2c30bebd9b4fb2f83bcc10ec9953
PageCount 4
ParticipantIDs ieee_primary_6787322
PublicationCentury 2000
PublicationDate 2014-Feb.
PublicationDateYYYYMMDD 2014-02-01
PublicationDate_xml – month: 02
  year: 2014
  text: 2014-Feb.
PublicationDecade 2010
PublicationTitle Proceedings - Euromicro Workshop on Parallel and Distributed Processing
PublicationTitleAbbrev EMPDP
PublicationYear 2014
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0007429
ssib026764338
Score 1.5383644
Snippet The performance of the Hadoop Distributed File System (HDFS)decreases dramatically when handling interaction-intensive files, i.e., files that have relatively...
SourceID ieee
SourceType Publisher
StartPage 508
SubjectTerms Arrays
Cache
HDFS
Hierarchical structure
Multi-layer neural network
Periodic structures
PSO
Resource management
Storage Allocation Algorithm
Throughput
Vectors
Title Enhancing Throughput of Hadoop Distributed File System for Interaction-Intensive Tasks
URI https://ieeexplore.ieee.org/document/6787322
WOSCitedRecordID wos000353964700076&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwED61FQNTgRbxlgdGTNOkdeKZtmJAVYaCulV-nKFCSqo--P2cnTYIiYXNiTJEPvse9n3fB3CPSaYTtJqjQskHqi_JD1rFs9hRgqENRjIAhV_S6TSbz2XegIcaC4OIofkMH_0w3OXb0uz8UVmPHGtKC7AJzTQVFVbrsHZikVJsTX6Ywqnkk-GmUwjui4S66V328lHue7oG_umXqEqIKZP2__7mBLo_4DyW12HnFBpYnEH7oM7A9pu1A2_j4sOTaRTvbFaJ8dAHrHSMnE1ZrtjIU-Z6tSu0bEK-gVXk5YyyWBbOCSvIA6-b3NlMbT43XXidjGdPz3yvosCXlBpsuZJRglqhzfoqQhfFSsggPEbJDg2VoKLExSaJNGor9cDp2GWJNqYfoZFymJxDqygLvADm0iFZnBImLQNiVgmPVJXSern62EaX0PHTtFhVRBmL_Qxd_f36Go69FaoW6Btobdc7vIUj87VdbtZ3wbrfY8yk4Q
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwED6VggRTgRbxxgMjpmmSJvFMWxVRqgwBdavs-AIVUlL1we_n7LSpkFjYnChD5LPvYd_3fQD36EXKQ604ShTclx1BflBLHrkZJRgqRUdYoPAoHI-jyUTENXiosDCIaJvP8NEM7V2-LtK1OSprk2MNaQHuwX7X912nRGttV48bhBRdvR1XOBV9wt51BgE3ZULV9i7acS82XV2-efolq2KjyqDxv_85htYOnsfiKvCcQA3zU2hs9RnYZrs24b2ffxo6jfyDJaUcD33AioyRuymKOesZ0lyjd4WaDcg7sJK-nFEey-xJYQl64FWbO0vk8mvZgrdBP3ka8o2OAp9RcrDiUjgeKok66kgHM8eVgbDSY5Tu0FAGVJZkbuo5CpUWys-Um0WeStOOg6kQXe8M6nmR4zmwLOySzSllUsJiZmVgsKpCaCNY72rnAppmmqbzkipjupmhy79f38HhMHkdTUfP45crODIWKRuir6G-WqzxBg7S79Vsubi1lv4BKGuoKA
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+-+Euromicro+Workshop+on+Parallel+and+Distributed+Processing&rft.atitle=Enhancing+Throughput+of+Hadoop+Distributed+File+System+for+Interaction-Intensive+Tasks&rft.au=Xiayu+Hua&rft.au=Hao+Wu&rft.au=Shangping+Ren&rft.date=2014-02-01&rft.pub=IEEE&rft.issn=1066-6192&rft.spage=508&rft.epage=511&rft_id=info:doi/10.1109%2FPDP.2014.110&rft.externalDocID=6787322
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1066-6192&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1066-6192&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1066-6192&client=summon