LogECMem: Coupling Erasure-Coded In-Memory Key-Value Stores with Parity Logging

In-memory key-value stores are often used to speed up Big Data workloads on modern HPC clusters. To maintain their high availability, erasure coding has been recently adopted as a low-cost redundancy scheme instead of replication. Existing erasure-coded update schemes, however, have either low perfo...

Full description

Saved in:
Bibliographic Details
Published in:SC21: International Conference for High Performance Computing, Networking, Storage and Analysis pp. 01 - 14
Main Authors: Cheng, Liangfeng, Hu, Yuchong, Ke, Zhaokang, Xu, Jia, Yao, Qiaori, Feng, Dan, Wang, Weichun, Chen, Wei
Format: Conference Proceeding
Language:English
Published: ACM 14.11.2021
Subjects:
Costs
Couplings
Erasure coding
High performance computing
Key-value stores
Maintenance engineering
Memory management
Parity logging
Prototypes
Redundancy
Update
ISSN:2167-4337
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract In-memory key-value stores are often used to speed up Big Data workloads on modern HPC clusters. To maintain their high availability, erasure coding has been recently adopted as a low-cost redundancy scheme instead of replication. Existing erasure-coded update schemes, however, have either low performance or high memory overhead. In this paper, we propose a novel parity logging-based architecture, HybridPL, which creates a hybrid of in-place update (for data and XOR parity chunks) and log-based update (for the remaining parity chunks), so as to balance the update performance and memory cost, while maintaining efficient single-failure repairs. We realize HybridPL as an in-memory key-value store called LogECMem, and further design efficient repair schemes for multiple failures. We prototype LogECMem and conduct experiments on different workloads. We show that LogECMem achieves better up-date performance over existing erasure-coded update schemes with low memory overhead, while maintaining high basic I/O and repair performance.
AbstractList In-memory key-value stores are often used to speed up Big Data workloads on modern HPC clusters. To maintain their high availability, erasure coding has been recently adopted as a low-cost redundancy scheme instead of replication. Existing erasure-coded update schemes, however, have either low performance or high memory overhead. In this paper, we propose a novel parity logging-based architecture, HybridPL, which creates a hybrid of in-place update (for data and XOR parity chunks) and log-based update (for the remaining parity chunks), so as to balance the update performance and memory cost, while maintaining efficient single-failure repairs. We realize HybridPL as an in-memory key-value store called LogECMem, and further design efficient repair schemes for multiple failures. We prototype LogECMem and conduct experiments on different workloads. We show that LogECMem achieves better up-date performance over existing erasure-coded update schemes with low memory overhead, while maintaining high basic I/O and repair performance.
Author Ke, Zhaokang
Chen, Wei
Yao, Qiaori
Cheng, Liangfeng
Xu, Jia
Hu, Yuchong
Feng, Dan
Wang, Weichun
Author_xml – sequence: 1
  givenname: Liangfeng
  surname: Cheng
  fullname: Cheng, Liangfeng
  email: lenfungcheng@hust.edu.cn
  organization: Huazhong University of Science and Technology,Wuhan,China
– sequence: 2
  givenname: Yuchong
  surname: Hu
  fullname: Hu, Yuchong
  email: yuchonghu@hust.edu.cn
  organization: Huazhong University of Science and Technology,Wuhan,China
– sequence: 3
  givenname: Zhaokang
  surname: Ke
  fullname: Ke, Zhaokang
  email: 2020612167@hust.edu.cn
  organization: Huazhong University of Science and Technology,Wuhan,China
– sequence: 4
  givenname: Jia
  surname: Xu
  fullname: Xu, Jia
  email: helenxu@hust.edu.cn
  organization: Huazhong University of Science and Technology,Wuhan,China
– sequence: 5
  givenname: Qiaori
  surname: Yao
  fullname: Yao, Qiaori
  email: yaoqr@hust.edu.cn
  organization: Huazhong University of Science and Technology,Wuhan,China
– sequence: 6
  givenname: Dan
  surname: Feng
  fullname: Feng, Dan
  email: dfeng@hust.edu.cn
  organization: Huazhong University of Science and Technology,Wuhan,China
– sequence: 7
  givenname: Weichun
  surname: Wang
  fullname: Wang, Weichun
  email: wangweichun@hikvision.com
  organization: HIKVISION,Hangzhou,China
– sequence: 8
  givenname: Wei
  surname: Chen
  fullname: Chen, Wei
  email: chenwei7@hikvision.com
  organization: HIKVISION,Hangzhou,China
BookMark eNotjMtKw0AUQEdRsNasXbiZH5h678xkHu4kVC1GKvjYlqRzUyNtUiYJkr83oKuzOJxzyc6atiHGrhEWiDq9VTp1Du1CaQculScs8dZNApTTWuIpm0k0Vmil7AVLuu4bAKSzqCTM2Dpvd8vshQ53PGuH475udnwZi26IJLI2UOCrRky6jSN_plF8FvuB-FvfRur4T91_8dci1v3Ip89uiq_YeVXsO0r-OWcfD8v37Enk68dVdp-LQtq0Fwq03XodUgALIaiydNqVRoOByuMWfTB6S6ayVKKnVGGobCidKTXpylRKzdnN37cmos0x1ocijhvvEVBa9Qv9BU94
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1145/3458817.3480852
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Xplore POP ALL
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISBN 9781450384421
1450384420
EISSN 2167-4337
EndPage 14
ExternalDocumentID 9910127
Genre orig-research
GrantInformation_xml – fundername: National Natural Science Foundation of China
  grantid: 61872414
  funderid: 10.13039/501100001809
GroupedDBID 6IE
6IF
6IH
6IK
6IL
6IN
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IPLJI
OCL
RIE
RIL
ID FETCH-LOGICAL-a275t-3047c94d50070dd3bb848b64060f91c19d64ce6f7eb19e531df7db86b4e4f6f33
IEDL.DBID RIE
ISICitedReferencesCount 12
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000946520100087&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:18:35 EDT 2025
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a275t-3047c94d50070dd3bb848b64060f91c19d64ce6f7eb19e531df7db86b4e4f6f33
PageCount 14
ParticipantIDs ieee_primary_9910127
PublicationCentury 2000
PublicationDate 2021-Nov.-14
PublicationDateYYYYMMDD 2021-11-14
PublicationDate_xml – month: 11
  year: 2021
  text: 2021-Nov.-14
  day: 14
PublicationDecade 2020
PublicationTitle SC21: International Conference for High Performance Computing, Networking, Storage and Analysis
PublicationTitleAbbrev SC
PublicationYear 2021
Publisher ACM
Publisher_xml – name: ACM
SSID ssj0002871320
ssj0003204180
Score 1.8563764
Snippet In-memory key-value stores are often used to speed up Big Data workloads on modern HPC clusters. To maintain their high availability, erasure coding has been...
SourceID ieee
SourceType Publisher
StartPage 01
SubjectTerms Costs
Couplings
Erasure coding
High performance computing
Key-value stores
Maintenance engineering
Memory management
Parity logging
Prototypes
Redundancy
Update
Title LogECMem: Coupling Erasure-Coded In-Memory Key-Value Stores with Parity Logging
URI https://ieeexplore.ieee.org/document/9910127
WOSCitedRecordID wos000946520100087&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwELbaioGpQIt4ywMjbpPYsR3WqBUIWirxULfKr1RIkKC2Qeq_x5eGIiQWNjuDZTm27_PdffchdOn3QWilEMTDAUaYYYZIIwOScBNrj7dVqFQlNiHGYzmdJpMGutpyYZxzVfKZ60GziuXbwpTgKut7LAOR0iZqCsE3XK2tPwWQP62hD_R9m4UyqKv5hCzuUyBlhqJHmfQ4I_olp1JZk2H7f_PYQ90fWh6ebA3OPmq4_AC1v3UZcH1MO-jhvpgP0pF7v8ZpUQLldo4Hi8oXSNLCOotvczKCFNs1vnNr8qLeSocf_evbLTE4ZvFEgaYd9uOAhlEXPQ8HT-kNqYUTiIpEvCIQSjMJszEU87GWai2Z1Nzb7iBLQhMmljPjeCb8RZ04fwptJqyWXDPHMp5ReohaeZG7I4SFiqiLIXyXJcxDO82pFpZpRT3SEAE7Rh1Yn9nHpjbGrF6ak78_n6LdCHJCII2OnaHWalG6c7RjPlevy8VF9UO_ALUgnd8
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LT8JAEN4gmugJFYxv9-DRhba77W69EoiEhySi4Ub2VWKirQFqwr93p1SMiRdv2z1Mmn10vs7MNx9Ct-4c-EZwThwcYIRpponQwiNxpEPl8Lb0pSzEJvhoJKbTeFxBd1sujLW2KD6zTRgWuXyT6RxCZS2HZSBTuoN2neHA27C1thEVwP60BD_w7MbMF17Zz8dnYYsCLdPnTcqEQxrBL0GVwp90a_97k0PU-CHm4fHW5Ryhik2PUe1bmQGXF7WOHgfZvNMe2vd73M5yIN3OcWdRRANJOzPW4F5KhlBku8Z9uyYv8i23-Mn9f9slhtAsHktQtcPODqgYNdBztzNpP5BSOoHIgIcrAsk0HTMTQjsfY6hSggkVOe_tJbGv_dhETNso4e5THVt3D03CjRKRYpYlUULpCaqmWWpPEeYyoDaEBF4SMwfuVEQVN0xJ6rAG99gZqsP6zD423TFm5dKc_z19g_YfJsPBbNAb9S_QQQAVIlBUxy5RdbXI7RXa05-r1-XiutjcLx4eoSY
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=SC21%3A+International+Conference+for+High+Performance+Computing%2C+Networking%2C+Storage+and+Analysis&rft.atitle=LogECMem%3A+Coupling+Erasure-Coded+In-Memory+Key-Value+Stores+with+Parity+Logging&rft.au=Cheng%2C+Liangfeng&rft.au=Hu%2C+Yuchong&rft.au=Ke%2C+Zhaokang&rft.au=Xu%2C+Jia&rft.date=2021-11-14&rft.pub=ACM&rft.eissn=2167-4337&rft.spage=01&rft.epage=14&rft_id=info:doi/10.1145%2F3458817.3480852&rft.externalDocID=9910127