Self-adaptive processing graph with operator fission for elastic stream processing

•Self adaptive mechanism for scaling stream processing systems.•Automatic scaling by increasing/decreasing the number of processing operators.•Model that changes graph topology based on a reactive and predictive algorithms.•Results show that both algorithms enable online self-adaptation of the graph...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:The Journal of systems and software Jg. 127; S. 205 - 216
Hauptverfasser: Hidalgo, Nicolas, Wladdimiro, Daniel, Rosas, Erika
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Elsevier Inc 01.05.2017
Schlagworte:
Elastic processing
Scalable processing
Self-adaptable graph
Stream processing
Scalable processing
S4
Stream processing
Self-adaptable graph
Elastic processing
ISSN:0164-1212, 1873-1228
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:•Self adaptive mechanism for scaling stream processing systems.•Automatic scaling by increasing/decreasing the number of processing operators.•Model that changes graph topology based on a reactive and predictive algorithms.•Results show that both algorithms enable online self-adaptation of the graph. Nowadays, information generated by the Internet interactions is growing exponentially, creating massive and continuous flows of events from the most diverse sources. These interactions contain valuable information for domains such as government, commerce, and banks, among others. Extracting information in near real-time from such data requires powerful processing tools to cope with the high-velocity and the high-volume stream of events. Specially designed distributed processing engines build a graph-based topology of a static number of processing operators creating bottlenecks and load balance problems when processing dynamic flows of events. In this work we propose a self-adaptive processing graph that provides elasticity and scalability by automatically increasing or decreasing the number of processing operators to improve performance and resource utilization of the system. Our solution uses a model that monitors, analyzes and changes the graph topology with a control algorithm that is both reactive and proactive to the flow of events. We have evaluated our solution with three stream processing applications and results show that our model can adapt the graph topology when receiving events at high rate with sudden peaks, producing very low costs of memory and CPU usage.
ISSN:0164-1212
1873-1228
DOI:10.1016/j.jss.2016.06.010