RDMA-Based Apache Storm for High-Performance Stream Data Processing

Apache Storm is a scalable fault-tolerant distributed real time stream-processing framework widely used in big data applications. For distributed data-sensitive applications, low-latency, high-throughput communication modules have a critical impact on overall system performance. Apache Storm current...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:International journal of parallel programming Ročník 49; číslo 5; s. 671 - 684
Hlavní autori: Zhang, Ziyu, Liu, Zitan, Jiang, Qingcai, Chen, Junshi, An, Hong
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: New York Springer US 01.10.2021
Springer Nature B.V
Predmet:
2020
Central processing units
Communication
Computer Science
Copying
CPUs
Data processing
Ethernet
Fault tolerance
Performance evaluation
Processor Architectures
Software Engineering/Programming and Operating Systems
Special Issue on Network and Parallel Computing for Emerging Architectures and Applications
TCP (protocol)
TCP/IP (protocol)
Technology assessment
Theory of Computation
RDMA
Apache Storm
Cloud computing
InfiniBand
Communication optimization
Stream-processing framework
ISSN:0885-7458, 1573-7640
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Apache Storm is a scalable fault-tolerant distributed real time stream-processing framework widely used in big data applications. For distributed data-sensitive applications, low-latency, high-throughput communication modules have a critical impact on overall system performance. Apache Storm currently uses Netty as its communication component, an asynchronous server/client framework based on TCP/IP protocol stack. The TCP/IP protocol stack has inherent performance flaws due to frequent memory copying and context switching. The Netty component not only limits the performance of the Storm but also increases the CPU load in the IPoIB (IP over InfiniBand) communication mode. In this paper, we introduce two new implementations for Apache Storm communication components with the help of RDMA technology. The performance evaluation on Mellanox QDR Cards (40 Gbps) shows that our implementations can achieve speedup up to 5 × compared with IPoIB and 10 × with Gigabit Ethernet. Our implementations also significantly reduce the CPU load and increase the throughput of the system.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0885-7458
1573-7640
DOI:10.1007/s10766-021-00696-0