Nime: a native in-memory compute framework for cluster computing - Chen, Chao; Wang, Zhenghua; Jiang, Chen; Wang, Zheng (Cluster computing, 2025)
Due to the increasing demand for cluster computing, various data analytics frameworks have been proposed and Apache Spark is a widely used open-source framework. It divides the program into various tasks and leverages executors on different machines for parallel task processing. However, executors run on top of Java virtual machines (JVMs), which incurs a significant runtime overhead in terms of memory and compute resources and thus deteriorates the system’s performance. In this paper, we present NIME–a native in-memory compute framework for cluster computing–that aims to perform parallel task processing using native executors. The key idea is that NIME starts off with native manager and worker processes without JVMs. In addition, a dedicated scheduler combines data partitions for efficient processing without interruptions and a cached is leveraged for iterative computations. We evaluate the effectiveness of NIME on a compute cluster using the HiBench benchmark suite and compare the results with those from the Spark framework. Evaluation results indicate that compared to Spark, on average NIME achieves a 6.82 × speedup, while simultaneously reducing the memory usage by 84.69%. In addition, the execution speedup and memory reduction can reach up to 12.36 × and 93.97%, respectively. Together with an in-depth analysis, we show that by discarding the JVM, NIME significantly accelerates task executions and minimizes the compute and memory resource overheads.