Nime: a native in-memory compute framework for cluster computing

Due to the increasing demand for cluster computing, various data analytics frameworks have been proposed and Apache Spark is a widely used open-source framework. It divides the program into various tasks and leverages executors on different machines for parallel task processing. However, executors r...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Cluster computing Jg. 28; H. 7; S. 434
Hauptverfasser: Chen, Chao, Wang, Zhenghua, Jiang, Chen, Wang, Zheng
Format: Journal Article
Sprache:Englisch
Veröffentlicht: New York Springer US 01.09.2025
Springer Nature B.V
Schlagworte:
Benchmarks
Breakdowns
Clusters
Compilers
Computation
Computer Communication Networks
Computer Science
Data processing
Datasets
Executors
Java
Machine learning
Operating Systems
Performance evaluation
Processor Architectures
Sanitation services
Virtual environments
In-memory computing
Spark
Distributed computing
Cluster computing
ISSN:1386-7857, 1573-7543
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Due to the increasing demand for cluster computing, various data analytics frameworks have been proposed and Apache Spark is a widely used open-source framework. It divides the program into various tasks and leverages executors on different machines for parallel task processing. However, executors run on top of Java virtual machines (JVMs), which incurs a significant runtime overhead in terms of memory and compute resources and thus deteriorates the system’s performance. In this paper, we present NIME–a native in-memory compute framework for cluster computing–that aims to perform parallel task processing using native executors. The key idea is that NIME starts off with native manager and worker processes without JVMs. In addition, a dedicated scheduler combines data partitions for efficient processing without interruptions and a cached is leveraged for iterative computations. We evaluate the effectiveness of NIME on a compute cluster using the HiBench benchmark suite and compare the results with those from the Spark framework. Evaluation results indicate that compared to Spark, on average NIME achieves a 6.82 × speedup, while simultaneously reducing the memory usage by 84.69%. In addition, the execution speedup and memory reduction can reach up to 12.36 × and 93.97%, respectively. Together with an in-depth analysis, we show that by discarding the JVM, NIME significantly accelerates task executions and minimizes the compute and memory resource overheads.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1386-7857
1573-7543
DOI:10.1007/s10586-025-05108-3