RocketHA: A High Availability Design Paradigm for Distributed Log-Based Storage System

As a team from Alibaba Cloud, we have developed and open-sourced RocketMQ, a cloud-native "messaging, eventing, streaming" real-time data processing platform that covers cloud-edge-device collaboration scenarios. During the development of RocketMQ, we also formulated RocketHA, a log-based...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE/ACM International Conference on Automated Software Engineering : [proceedings] s. 1819 - 1824
Hlavní autoři: Ji, Juntao, Jin, Rongtong, Fu, Yubao, Gu, Yinyou, Tsai, Tsung-han, Lin, Qingshan
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 11.09.2023
Témata:
automatic recovery
Collaboration
Computer crashes
Data processing
high-availability
log-based software
Real-time systems
Software
Software engineering
Throughput
ISSN:2643-1572
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:As a team from Alibaba Cloud, we have developed and open-sourced RocketMQ, a cloud-native "messaging, eventing, streaming" real-time data processing platform that covers cloud-edge-device collaboration scenarios. During the development of RocketMQ, we also formulated RocketHA, a log-based storage high availability design theory that provides a robust solution for distributed log storage software used in industrial applications. RocketHA comprises six fundamental components that enable automatic cluster recovery from failures such as crashes and partitions. This design paradigm has been successfully implemented in the open-source RocketMQ. Our evaluation demonstrates that RocketHA ensures high availability, fast recovery, high throughput, and data loss prevention. We hope that RocketHA will inspire and guide the development of high-availability solutions for all log-based storage systems.
ISSN:2643-1572
DOI:10.1109/ASE56229.2023.00123