RocketHA: A High Availability Design Paradigm for Distributed Log-Based Storage System

As a team from Alibaba Cloud, we have developed and open-sourced RocketMQ, a cloud-native "messaging, eventing, streaming" real-time data processing platform that covers cloud-edge-device collaboration scenarios. During the development of RocketMQ, we also formulated RocketHA, a log-based...

Full description

Saved in:
Bibliographic Details
Published in:IEEE/ACM International Conference on Automated Software Engineering : [proceedings] pp. 1819 - 1824
Main Authors: Ji, Juntao, Jin, Rongtong, Fu, Yubao, Gu, Yinyou, Tsai, Tsung-han, Lin, Qingshan
Format: Conference Proceeding
Language:English
Published: IEEE 11.09.2023
Subjects:
automatic recovery
Collaboration
Computer crashes
Data processing
high-availability
log-based software
Real-time systems
Software
Software engineering
Throughput
ISSN:2643-1572
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract As a team from Alibaba Cloud, we have developed and open-sourced RocketMQ, a cloud-native "messaging, eventing, streaming" real-time data processing platform that covers cloud-edge-device collaboration scenarios. During the development of RocketMQ, we also formulated RocketHA, a log-based storage high availability design theory that provides a robust solution for distributed log storage software used in industrial applications. RocketHA comprises six fundamental components that enable automatic cluster recovery from failures such as crashes and partitions. This design paradigm has been successfully implemented in the open-source RocketMQ. Our evaluation demonstrates that RocketHA ensures high availability, fast recovery, high throughput, and data loss prevention. We hope that RocketHA will inspire and guide the development of high-availability solutions for all log-based storage systems.
AbstractList As a team from Alibaba Cloud, we have developed and open-sourced RocketMQ, a cloud-native "messaging, eventing, streaming" real-time data processing platform that covers cloud-edge-device collaboration scenarios. During the development of RocketMQ, we also formulated RocketHA, a log-based storage high availability design theory that provides a robust solution for distributed log storage software used in industrial applications. RocketHA comprises six fundamental components that enable automatic cluster recovery from failures such as crashes and partitions. This design paradigm has been successfully implemented in the open-source RocketMQ. Our evaluation demonstrates that RocketHA ensures high availability, fast recovery, high throughput, and data loss prevention. We hope that RocketHA will inspire and guide the development of high-availability solutions for all log-based storage systems.
Author Ji, Juntao
Lin, Qingshan
Fu, Yubao
Gu, Yinyou
Jin, Rongtong
Tsai, Tsung-han
Author_xml – sequence: 1
  givenname: Juntao
  surname: Ji
  fullname: Ji, Juntao
  email: juntao.jjt@alibaba-inc.com
  organization: Alibaba Cloud Computing Co. Ltd.,Hangzhou,Zhejiang,China
– sequence: 2
  givenname: Rongtong
  surname: Jin
  fullname: Jin, Rongtong
  email: jinrongtong@apache.org
  organization: Alibaba Cloud Computing Co. Ltd.,Hangzhou,Zhejiang,China
– sequence: 3
  givenname: Yubao
  surname: Fu
  fullname: Fu, Yubao
  email: fuyou@apache.org
  organization: Alibaba Cloud Computing Co. Ltd.,Hangzhou,Zhejiang,China
– sequence: 4
  givenname: Yinyou
  surname: Gu
  fullname: Gu, Yinyou
  email: guyinyou.gyy@alibaba-inc.com
  organization: Alibaba Cloud Computing Co. Ltd.,Hangzhou,Zhejiang,China
– sequence: 5
  givenname: Tsung-han
  surname: Tsai
  fullname: Tsai, Tsung-han
  email: tsaitsunghan@apache.org
  organization: Alibaba Cloud Computing Co. Ltd.,Hangzhou,Zhejiang,China
– sequence: 6
  givenname: Qingshan
  surname: Lin
  fullname: Lin, Qingshan
  email: linhill@apache.org
  organization: Alibaba Cloud Computing Co. Ltd.,Hangzhou,Zhejiang,China
BookMark eNotj99KwzAchaMouM09gV7kBVrzp2ka7-o2rVBQrHo70uTXGu1aaTKhb29Br8754PDBWaKzfugBoStKYkqJusmrnUgZUzEjjMeEUMZP0FpJlXFBOFMqTU7RgqUJj6iQ7AItvf8kRMwgF-j9ZTBfEIr8Fue4cO0Hzn-063TtOhcmvAXv2h4_61Fb1x5wM4x463wYXX0MYHE5tNGd9nOrwjDqFnA1-QCHS3Te6M7D-j9X6O1-97opovLp4XGTl5FmWRIisCkYZZKEKGpSnVHFRcMtpcClqmtpMzC15FyJjDIh5oUxNBUEjEmstJKv0PWf1wHA_nt0Bz1Oe0rY_J4R_gtru1Kj
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/ASE56229.2023.00123
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISBN 9798350329964
EISSN 2643-1572
EndPage 1824
ExternalDocumentID 10298320
Genre orig-research
GroupedDBID 6IE
6IF
6IH
6IK
6IL
6IM
6IN
6J9
AAJGR
AAWTH
ABLEC
ACREN
ADYOE
ADZIZ
AFYQB
ALMA_UNASSIGNED_HOLDINGS
AMTXH
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IPLJI
M43
OCL
RIE
RIL
ID FETCH-LOGICAL-a284t-ed6ec9c44091c6a81935f3d11e379bb7d8ecb73395812556a8cc1650ecc4d7d73
IEDL.DBID RIE
ISICitedReferencesCount 0
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001103357200162&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:32:41 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a284t-ed6ec9c44091c6a81935f3d11e379bb7d8ecb73395812556a8cc1650ecc4d7d73
PageCount 6
ParticipantIDs ieee_primary_10298320
PublicationCentury 2000
PublicationDate 2023-Sept.-11
PublicationDateYYYYMMDD 2023-09-11
PublicationDate_xml – month: 09
  year: 2023
  text: 2023-Sept.-11
  day: 11
PublicationDecade 2020
PublicationTitle IEEE/ACM International Conference on Automated Software Engineering : [proceedings]
PublicationTitleAbbrev ASE
PublicationYear 2023
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0051577
ssib057256115
Score 2.2375169
Snippet As a team from Alibaba Cloud, we have developed and open-sourced RocketMQ, a cloud-native "messaging, eventing, streaming" real-time data processing platform...
SourceID ieee
SourceType Publisher
StartPage 1819
SubjectTerms automatic recovery
Collaboration
Computer crashes
Data processing
high-availability
log-based software
Real-time systems
Software
Software engineering
Throughput
Title RocketHA: A High Availability Design Paradigm for Distributed Log-Based Storage System
URI https://ieeexplore.ieee.org/document/10298320
WOSCitedRecordID wos001103357200162&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LTwIxEG6UePCED4zv9OC1SvfV1tsqEg6EEFHCjXTbgZAoGFxI_PfO7C7oxYO3TbOHptP2m-nM9w1jNwiBiMJRIjKnrYiczoS2pinAkBq5C70seGvDrur19Ghk-hVZveDCAEBRfAa39Fnk8v3CreipDE94YHAHYoS-q5QqyVqbzRMrBG8pt74v4rRSlcyQbJq7dPCEUB8QNyUgUVNJDYp-NVQp8KRd_-dMDljjh5nH-1vMOWQ7MD9i9U1rBl6d1GM2fMZ7DvJOes9TTrUcPF3b2Vspyv3FW0XdBu_bpfWz6TtHz5W3SEKXul-B593FVDwgvnk-wJgcrxxeKps32Gv76eWxI6oWCsIi7uQCfALOuAijOOkSi_AfxhMyAITKZJnyGlymwtDECPRxjH84J9FpQ8NGXnkVnrDafDGHU8YzjIz0JNY-SzylQym_58LIygAmifLxGWvQOo0_SpWM8WaJzv8Yv2D7ZAqqvZDyktXy5Qqu2J5b57PP5XVh229sjaJU
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NTwIxEG0MmugJPzB-24PXKt3dbrfeVsFgXAkRJNxIty2ERMHgQuK_d2Z3QS8evG2aPTSdtm-mM-8NIVcAgYDCQchSE2kWmChlkVZ15hSqkRvf8py31k9kux0NBqpTktVzLoxzLi8-c9f4mefy7cws8KkMTrinYAdChL4pgsDjBV1rtX2EBPjmfO39AlJLWQoN8bq6ibtNAHsP2SkeyppybFH0q6VKjigP1X_OZZfUfrh5tLNGnT2y4ab7pLpqzkDLs3pA-i9w07msFd_SmGI1B42XevJWyHJ_0UZeuUE7eq7tZPxOwXelDRTRxf5XztJkNmZ3gHCWdiEqh0uHFtrmNfL60Ozdt1jZRIFpQJ6MORs6o0wAcRw3oQYHwBcjNIHzpUpTaSNnUun7SgDUCwF_GMPBbQPTBlZa6R-SynQ2dUeEphAbRSMR2TS0mBDFDJ_xA809NwqlFcekhus0_Ch0MoarJTr5Y_ySbLd6z8kweWw_nZIdNAtWYnB-RirZfOHOyZZZZpPP-UVu528PLKWb
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=IEEE%2FACM+International+Conference+on+Automated+Software+Engineering+%3A+%5Bproceedings%5D&rft.atitle=RocketHA%3A+A+High+Availability+Design+Paradigm+for+Distributed+Log-Based+Storage+System&rft.au=Ji%2C+Juntao&rft.au=Jin%2C+Rongtong&rft.au=Fu%2C+Yubao&rft.au=Gu%2C+Yinyou&rft.date=2023-09-11&rft.pub=IEEE&rft.eissn=2643-1572&rft.spage=1819&rft.epage=1824&rft_id=info:doi/10.1109%2FASE56229.2023.00123&rft.externalDocID=10298320