View in EDS

Intelligent Hybrid Caching for Sustainable Big Data Processing: Leveraging NVM to Enable Green Digital Transformation.

Saved in:
Bibliographic Details
Title: Intelligent Hybrid Caching for Sustainable Big Data Processing: Leveraging NVM to Enable Green Digital Transformation.
Authors: Tong, Lei, Shen, Qing, Xie, Zhenqiang
Source: Sustainability (2071-1050); Mar2026, Vol. 18 Issue 5, p2601, 23p
Abstract: Apache Spark has gained widespread adoption for large-scale data processing. However, conventional caching methods inadequately address the dual challenges of performance bottlenecks and escalating energy consumption in data-intensive workloads. This paper introduces a sustainable computing framework that integrates Directed Acyclic Graph (DAG) dependency analysis with garbage collection (GC) behavior monitoring to optimize data placement between DRAM and non-volatile memory (NVM). The proposed Intelligent Hybrid Caching Management Framework (IHCMF) dynamically predicts data access patterns and migrates cache blocks based on cost–benefit analysis, achieving a 37.5% execution time reduction over default Spark configurations in SparkBench evaluations. By improving throughput-per-watt and projecting potential benefits from NVM's near-zero idle power and extended hardware lifespan, IHCMF provides a scalable, cost-effective caching solution for resource-constrained edge computing environments. This work demonstrates that high-performance computing can be reconciled with environmental sustainability through intelligent memory management. [ABSTRACT FROM AUTHOR]
Copyright of Sustainability (2071-1050) is the property of MDPI and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Complementary Index
Description
Abstract:Apache Spark has gained widespread adoption for large-scale data processing. However, conventional caching methods inadequately address the dual challenges of performance bottlenecks and escalating energy consumption in data-intensive workloads. This paper introduces a sustainable computing framework that integrates Directed Acyclic Graph (DAG) dependency analysis with garbage collection (GC) behavior monitoring to optimize data placement between DRAM and non-volatile memory (NVM). The proposed Intelligent Hybrid Caching Management Framework (IHCMF) dynamically predicts data access patterns and migrates cache blocks based on cost–benefit analysis, achieving a 37.5% execution time reduction over default Spark configurations in SparkBench evaluations. By improving throughput-per-watt and projecting potential benefits from NVM's near-zero idle power and extended hardware lifespan, IHCMF provides a scalable, cost-effective caching solution for resource-constrained edge computing environments. This work demonstrates that high-performance computing can be reconciled with environmental sustainability through intelligent memory management. [ABSTRACT FROM AUTHOR]
ISSN:20711050
DOI:10.3390/su18052601