In EDS ansehen

Reinforcement Learning Hadoop Map Reduce Parameters Optimization

Gespeichert in:
Bibliographische Detailangaben
Titel: Reinforcement Learning Hadoop Map Reduce Parameters Optimization
Autoren: Nandita Yambem
Quelle: International Journal of Intelligent Systems and Applications in Engineering; Vol. 13 No. 1 (2025); 63 – 69
Verlagsinformationen: International Journal of Intelligent Systems and Applications in Engineering, 2025.
Publikationsjahr: 2025
Schlagwörter: Hadoop, reinforcement learning, q-learning, mapreduce, HDFS
Beschreibung: Among the various techniques for enhancing Hadoop performance—such as intermediate data compression, in-memory management, and parameter tuning—dynamic configuration parameter tuning proves to be the most impactful. However, existing approaches face several challenges: limited adaptability to specific application requirements, isolated parameter tuning without considering interdependencies, and inaccurate linear assumptions in complex environments. To address these issues, this study introduces a reinforcement learning-based optimization framework using Q-Learning. The proposed method dynamically adjusts key Hadoop configuration parameters by continuously learning from job execution metrics such as completion time and wait times in map/reduce phases. It employs a reward-based feedback mechanism to minimize the gap between expected and actual performance, ensuring more accurate, adaptive, and holistic optimization. Additionally, the framework integrates a neural network to predict optimal parameter values, further enhancing decision-making. This approach significantly improves execution efficiency and resource utilization, offering robust adaptability across diverse workloads and operational environments, while aligning closely with service level agreements.
Publikationsart: Article
Dateibeschreibung: application/pdf
Sprache: English
ISSN: 2147-6799
Zugangs-URL: https://www.ijisae.org/index.php/IJISAE/article/view/7448
Rights: CC BY SA
Dokumentencode: edsair.issn21476799..a28de2e41ebdc16f277491f4f7da6bf7
Datenbank: OpenAIRE
Beschreibung
Abstract:Among the various techniques for enhancing Hadoop performance—such as intermediate data compression, in-memory management, and parameter tuning—dynamic configuration parameter tuning proves to be the most impactful. However, existing approaches face several challenges: limited adaptability to specific application requirements, isolated parameter tuning without considering interdependencies, and inaccurate linear assumptions in complex environments. To address these issues, this study introduces a reinforcement learning-based optimization framework using Q-Learning. The proposed method dynamically adjusts key Hadoop configuration parameters by continuously learning from job execution metrics such as completion time and wait times in map/reduce phases. It employs a reward-based feedback mechanism to minimize the gap between expected and actual performance, ensuring more accurate, adaptive, and holistic optimization. Additionally, the framework integrates a neural network to predict optimal parameter values, further enhancing decision-making. This approach significantly improves execution efficiency and resource utilization, offering robust adaptability across diverse workloads and operational environments, while aligning closely with service level agreements.
ISSN:21476799