Benchmarking Streaming Computation Engines: Storm, Flink and Spark Streaming

Streaming data processing has been gaining attention due to its application into a wide range of scenarios. To serve the booming demands of streaming data processing, many computation engines have been developed. However, there is still a lack of real-world benchmarks that would be helpful when choo...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) S. 1789 - 1792
Hauptverfasser: Chintapalli, Sanket, Dagit, Derek, Evans, Bobby, Farivar, Reza, Graves, Thomas, Holderbaugh, Mark, Zhuo Liu, Nusbaum, Kyle, Patil, Kishorkumar, Peng, Boyang Jerry, Poulosky, Paul
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: IEEE 01.05.2016
Schlagworte:
Benchmark
Benchmark testing
Data processing
Engines
Flink
Low Latency
Pipelines
Spark
Sparks
Storm
Storms
Streaming processing
Throughput
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Streaming data processing has been gaining attention due to its application into a wide range of scenarios. To serve the booming demands of streaming data processing, many computation engines have been developed. However, there is still a lack of real-world benchmarks that would be helpful when choosing the most appropriate platform for serving real-time streaming needs. In order to address this problem, we developed a streaming benchmark for three representative computation engines: Flink, Storm and Spark Streaming. Instead of testing speed-of-light event processing, we construct a full data pipeline using Kafka and Redis in order to more closely mimic the real-world production scenarios. Based on our experiments, we provide a performance comparison of the three data engines in terms of 99th percentile latency and throughput for various configurations.
DOI:10.1109/IPDPSW.2016.138