Nime: a native in-memory compute framework for cluster computing

Due to the increasing demand for cluster computing, various data analytics frameworks have been proposed and Apache Spark is a widely used open-source framework. It divides the program into various tasks and leverages executors on different machines for parallel task processing. However, executors r...

Full description

Saved in:
Bibliographic Details
Published in:Cluster computing Vol. 28; no. 7; p. 434
Main Authors: Chen, Chao, Wang, Zhenghua, Jiang, Chen, Wang, Zheng
Format: Journal Article
Language:English
Published: New York Springer US 01.09.2025
Springer Nature B.V
Subjects:
Benchmarks
Breakdowns
Clusters
Compilers
Computation
Computer Communication Networks
Computer Science
Data processing
Datasets
Executors
Java
Machine learning
Operating Systems
Performance evaluation
Processor Architectures
Sanitation services
Virtual environments
In-memory computing
Spark
Distributed computing
Cluster computing
ISSN:1386-7857, 1573-7543
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Due to the increasing demand for cluster computing, various data analytics frameworks have been proposed and Apache Spark is a widely used open-source framework. It divides the program into various tasks and leverages executors on different machines for parallel task processing. However, executors run on top of Java virtual machines (JVMs), which incurs a significant runtime overhead in terms of memory and compute resources and thus deteriorates the system’s performance. In this paper, we present NIME–a native in-memory compute framework for cluster computing–that aims to perform parallel task processing using native executors. The key idea is that NIME starts off with native manager and worker processes without JVMs. In addition, a dedicated scheduler combines data partitions for efficient processing without interruptions and a cached is leveraged for iterative computations. We evaluate the effectiveness of NIME on a compute cluster using the HiBench benchmark suite and compare the results with those from the Spark framework. Evaluation results indicate that compared to Spark, on average NIME achieves a 6.82 × speedup, while simultaneously reducing the memory usage by 84.69%. In addition, the execution speedup and memory reduction can reach up to 12.36 × and 93.97%, respectively. Together with an in-depth analysis, we show that by discarding the JVM, NIME significantly accelerates task executions and minimizes the compute and memory resource overheads.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1386-7857
1573-7543
DOI:10.1007/s10586-025-05108-3