StarData: Distributed Map-Reduce Framework Atop Serverless Computing Platforms

With the rapid development of information networks, intelligent terminals, the Internet of Things, and various sensing devices continue to expand. A large amount of isomorphic and heterogeneous data has been generated. How to process these data efficiently and in real time has become the leading res...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:IEEE access Ročník 13; s. 182633 - 182644
Hlavní autori: Zhou, Shuhui, He, Xing, Zhang, Zhenshuo, Gao, Feng, Hong, Yuan, Gao, Ruijin, Wang, Xiaoming, Yun, Hongsheng, Ma, Ruhui, Sun, Jinshan
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Piscataway IEEE 2025
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Predmet:
AWS Lambda
Big Data
cloud computing
Cost effectiveness
Costs
Data analysis
Data processing
Distributed databases
Internet of Things
map-reduce
Parallel processing
Python
Random access memory
Resource allocation
Scalability
Sensors
Serverless computing
Sun
ISSN:2169-3536, 2169-3536
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:With the rapid development of information networks, intelligent terminals, the Internet of Things, and various sensing devices continue to expand. A large amount of isomorphic and heterogeneous data has been generated. How to process these data efficiently and in real time has become the leading research direction of extensive data analysis and processing. However, the traditional big data processing platform requires help to meet the basic requirements for efficiently processing large amounts of data quickly. The Map-Reduce extensive data processing framework proposed by Google can process various types of homogeneous and heterogeneous data. Users can analyze and process data in the computing cluster after configuring it as needed. We propose StarData. It is an extensive data processing framework based on a service-oriented computing platform. By building the Map-Reduce processing framework on AWS Lambda, a serverless computing platform, we can realize the requirement of parallel processing of large-scale data in a short time. StarData is a lightweight extensive data analysis and processing framework with good cost-effectiveness. The efficiency of comprehensive data analysis and processing is optimized by reasonably adjusting the number of functions, files, file size, and resource allocation.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2025.3621096