DSParLib: A C++ Template Library for Distributed Stream Parallelism

Stream processing applications deal with millions of data items continuously generated over time. Often, they must be processed in real-time and scale performance, which requires the use of distributed parallel computing resources. In C/C++, the current state-of-the-art for distributed architectures...

Full description

Saved in:
Bibliographic Details
Published in:International journal of parallel programming Vol. 50; no. 5-6; pp. 454 - 485
Main Authors: Löff, Júnior, Hoffmann, Renato B., Pieper, Ricardo, Griebler, Dalvan, Fernandes, Luiz G.
Format: Journal Article
Language:English
Published: New York Springer US 01.12.2022
Springer Nature B.V
Subjects:
Big Data
C plus plus
C++ (programming language)
Communication
Computer Science
Data transmission
Distributed processing
Experiments
Handwriting
Interfaces
Java
Message passing
Parallel processing
Parallel programming
Processor Architectures
Programmers
Programming languages
Software Engineering/Programming and Operating Systems
Special Issue on High-Level Parallel Programming and Applications 2021
Theory of Computation
Stream processing
Parallel programming
Distributed systems
Parallel patterns
MPI
ISSN:0885-7458, 1573-7640
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Stream processing applications deal with millions of data items continuously generated over time. Often, they must be processed in real-time and scale performance, which requires the use of distributed parallel computing resources. In C/C++, the current state-of-the-art for distributed architectures and High-Performance Computing is Message Passing Interface (MPI). However, exploiting stream parallelism using MPI is complex and error-prone because it exposes many low-level details to the programmer. In this work, we introduce a new parallel programming abstraction for implementing distributed stream parallelism named DSParLib. Our abstraction of MPI simplifies parallel programming by providing a pattern-based and building block-oriented development to inter-connect, model, and parallelize data streams found in modern applications. Experiments conducted with five different stream processing applications and the representative PARSEC Ferret benchmark revealed that DSParLib is efficient and flexible. Also, DSParLib achieved similar or better performance, required less coding, and provided simpler abstractions to express parallelism with respect to handwritten MPI programs.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0885-7458
1573-7640
DOI:10.1007/s10766-022-00737-2