StarData: Distributed Map-Reduce Framework Atop Serverless Computing Platforms

With the rapid development of information networks, intelligent terminals, the Internet of Things, and various sensing devices continue to expand. A large amount of isomorphic and heterogeneous data has been generated. How to process these data efficiently and in real time has become the leading res...

Full description

Saved in:
Bibliographic Details
Published in:IEEE access Vol. 13; pp. 182633 - 182644
Main Authors: Zhou, Shuhui, He, Xing, Zhang, Zhenshuo, Gao, Feng, Hong, Yuan, Gao, Ruijin, Wang, Xiaoming, Yun, Hongsheng, Ma, Ruhui, Sun, Jinshan
Format: Journal Article
Language:English
Published: Piscataway IEEE 2025
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
AWS Lambda
Big Data
cloud computing
Cost effectiveness
Costs
Data analysis
Data processing
Distributed databases
Internet of Things
map-reduce
Parallel processing
Python
Random access memory
Resource allocation
Scalability
Sensors
Serverless computing
Sun
ISSN:2169-3536, 2169-3536
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:With the rapid development of information networks, intelligent terminals, the Internet of Things, and various sensing devices continue to expand. A large amount of isomorphic and heterogeneous data has been generated. How to process these data efficiently and in real time has become the leading research direction of extensive data analysis and processing. However, the traditional big data processing platform requires help to meet the basic requirements for efficiently processing large amounts of data quickly. The Map-Reduce extensive data processing framework proposed by Google can process various types of homogeneous and heterogeneous data. Users can analyze and process data in the computing cluster after configuring it as needed. We propose StarData. It is an extensive data processing framework based on a service-oriented computing platform. By building the Map-Reduce processing framework on AWS Lambda, a serverless computing platform, we can realize the requirement of parallel processing of large-scale data in a short time. StarData is a lightweight extensive data analysis and processing framework with good cost-effectiveness. The efficiency of comprehensive data analysis and processing is optimized by reasonably adjusting the number of functions, files, file size, and resource allocation.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2025.3621096