HCL: Distributing Parallel Data Structures in Extreme Scales

Most parallel programs use irregular control flow and data structures, which are perfect for one-sided communication paradigms such as MPI or PGAS programming languages. However, these environments lack the presence of efficient function-based application libraries that can utilize popular communica...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings / IEEE International Conference on Cluster Computing pp. 248 - 258
Main Authors: Devarajan, Hariharan, Kougkas, Anthony, Bateman, Keith, Sun, Xian-He
Format: Conference Proceeding
Language:English
Published: IEEE 01.09.2020
Subjects:
Computational modeling
Containers
Data models
Data structures
Distributed Data Structure
Distributed databases
HPC Data Containers
Hybrid Data Access Model
Libraries
Programming
RPC over RDMA
ISSN:2168-9253
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Most parallel programs use irregular control flow and data structures, which are perfect for one-sided communication paradigms such as MPI or PGAS programming languages. However, these environments lack the presence of efficient function-based application libraries that can utilize popular communication fabrics such as TCP, Infinity Band (IB), and RDMA over Converged Ethernet (RoCE). Additionally, there is a lack of high-performance data structure interfaces. We present Hermes Container Library (HCL), a high-performance distributed data structures library that offers high-level abstractions including hash-maps, sets, and queues. HCL uses a RPC over RDMA technology that implements a novel procedural programming paradigm. In this paper, we argue a RPC over RDMA technology can serve as a high-performance, flexible, and co-ordination free backend for implementing complex data structures. Evaluation results from testing real workloads shows that HCL programs are 2x to 12x faster compared to BCL, a state-of-the-art distributed data structure library.
ISSN:2168-9253
DOI:10.1109/CLUSTER49012.2020.00035