Zobrazit v EDS

Nime: a native in-memory compute framework for cluster computing.

Uloženo v:
Podrobná bibliografie
Název: Nime: a native in-memory compute framework for cluster computing.
Autoři: Chen, Chao, Wang, Zhenghua, Jiang, Chen, Wang, Zheng
Zdroj: Cluster Computing; Oct2025, Vol. 28 Issue 7, p1-19, 19p
Témata: PARALLEL processing, HIGH performance computing, COMPUTER workstation clusters, ELECTRONIC data processing, COMPUTER performance
Abstrakt: Due to the increasing demand for cluster computing, various data analytics frameworks have been proposed and Apache Spark is a widely used open-source framework. It divides the program into various tasks and leverages executors on different machines for parallel task processing. However, executors run on top of Java virtual machines (JVMs), which incurs a significant runtime overhead in terms of memory and compute resources and thus deteriorates the system's performance. In this paper, we present NIME–a native in-memory compute framework for cluster computing–that aims to perform parallel task processing using native executors. The key idea is that NIME starts off with native manager and worker processes without JVMs. In addition, a dedicated scheduler combines data partitions for efficient processing without interruptions and a cached is leveraged for iterative computations. We evaluate the effectiveness of NIME on a compute cluster using the HiBench benchmark suite and compare the results with those from the Spark framework. Evaluation results indicate that compared to Spark, on average NIME achieves a 6.82 × speedup, while simultaneously reducing the memory usage by 84.69%. In addition, the execution speedup and memory reduction can reach up to 12.36 × and 93.97%, respectively. Together with an in-depth analysis, we show that by discarding the JVM, NIME significantly accelerates task executions and minimizes the compute and memory resource overheads. [ABSTRACT FROM AUTHOR]
Copyright of Cluster Computing is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Databáze: Complementary Index
Popis
Abstrakt:Due to the increasing demand for cluster computing, various data analytics frameworks have been proposed and Apache Spark is a widely used open-source framework. It divides the program into various tasks and leverages executors on different machines for parallel task processing. However, executors run on top of Java virtual machines (JVMs), which incurs a significant runtime overhead in terms of memory and compute resources and thus deteriorates the system's performance. In this paper, we present NIME–a native in-memory compute framework for cluster computing–that aims to perform parallel task processing using native executors. The key idea is that NIME starts off with native manager and worker processes without JVMs. In addition, a dedicated scheduler combines data partitions for efficient processing without interruptions and a cached is leveraged for iterative computations. We evaluate the effectiveness of NIME on a compute cluster using the HiBench benchmark suite and compare the results with those from the Spark framework. Evaluation results indicate that compared to Spark, on average NIME achieves a 6.82 × speedup, while simultaneously reducing the memory usage by 84.69%. In addition, the execution speedup and memory reduction can reach up to 12.36 × and 93.97%, respectively. Together with an in-depth analysis, we show that by discarding the JVM, NIME significantly accelerates task executions and minimizes the compute and memory resource overheads. [ABSTRACT FROM AUTHOR]
ISSN:13867857
DOI:10.1007/s10586-025-05108-3