Benchmarking Streaming Computation Engines: Storm, Flink and Spark Streaming

Streaming data processing has been gaining attention due to its application into a wide range of scenarios. To serve the booming demands of streaming data processing, many computation engines have been developed. However, there is still a lack of real-world benchmarks that would be helpful when choo...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) s. 1789 - 1792
Hlavní autoři: Chintapalli, Sanket, Dagit, Derek, Evans, Bobby, Farivar, Reza, Graves, Thomas, Holderbaugh, Mark, Zhuo Liu, Nusbaum, Kyle, Patil, Kishorkumar, Peng, Boyang Jerry, Poulosky, Paul
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 01.05.2016
Témata:
Benchmark
Benchmark testing
Data processing
Engines
Flink
Low Latency
Pipelines
Spark
Sparks
Storm
Storms
Streaming processing
Throughput
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Streaming data processing has been gaining attention due to its application into a wide range of scenarios. To serve the booming demands of streaming data processing, many computation engines have been developed. However, there is still a lack of real-world benchmarks that would be helpful when choosing the most appropriate platform for serving real-time streaming needs. In order to address this problem, we developed a streaming benchmark for three representative computation engines: Flink, Storm and Spark Streaming. Instead of testing speed-of-light event processing, we construct a full data pipeline using Kafka and Redis in order to more closely mimic the real-world production scenarios. Based on our experiments, we provide a performance comparison of the three data engines in terms of 99th percentile latency and throughput for various configurations.
DOI:10.1109/IPDPSW.2016.138