LogECMem: Coupling Erasure-Coded In-Memory Key-Value Stores with Parity Logging

In-memory key-value stores are often used to speed up Big Data workloads on modern HPC clusters. To maintain their high availability, erasure coding has been recently adopted as a low-cost redundancy scheme instead of replication. Existing erasure-coded update schemes, however, have either low perfo...

Full description

Saved in:
Bibliographic Details
Published in:SC21: International Conference for High Performance Computing, Networking, Storage and Analysis pp. 01 - 14
Main Authors: Cheng, Liangfeng, Hu, Yuchong, Ke, Zhaokang, Xu, Jia, Yao, Qiaori, Feng, Dan, Wang, Weichun, Chen, Wei
Format: Conference Proceeding
Language:English
Published: ACM 14.11.2021
Subjects:
Costs
Couplings
Erasure coding
High performance computing
Key-value stores
Maintenance engineering
Memory management
Parity logging
Prototypes
Redundancy
Update
ISSN:2167-4337
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In-memory key-value stores are often used to speed up Big Data workloads on modern HPC clusters. To maintain their high availability, erasure coding has been recently adopted as a low-cost redundancy scheme instead of replication. Existing erasure-coded update schemes, however, have either low performance or high memory overhead. In this paper, we propose a novel parity logging-based architecture, HybridPL, which creates a hybrid of in-place update (for data and XOR parity chunks) and log-based update (for the remaining parity chunks), so as to balance the update performance and memory cost, while maintaining efficient single-failure repairs. We realize HybridPL as an in-memory key-value store called LogECMem, and further design efficient repair schemes for multiple failures. We prototype LogECMem and conduct experiments on different workloads. We show that LogECMem achieves better up-date performance over existing erasure-coded update schemes with low memory overhead, while maintaining high basic I/O and repair performance.
ISSN:2167-4337
DOI:10.1145/3458817.3480852