A hierarchical reinforcement learning-aware hyper-heuristic algorithm with fitness landscape analysis - Zhu, Ningning; Zhao, Fuqing; Yu, Yang; Wang, Ling (Swarm and evolutionary computation, 2024)
•A hyperheuristic algorithm with hierarchical reinforcement learning is proposed.•Two fitness landscape analyses as high-level control strategies are designed.•The automatic algorithm design based on multiple meta-heuristics is presented.•The rewards in light of the real-time dynamic environment are introduced.•Three action selection strategies based on problem characteristics are employed. The automation of meta-heuristic algorithm configuration holds the utmost significance in evolutionary computation. A hierarchical reinforcement learning-aware hyper-heuristic algorithm with fitness landscape analysis (HRLHH) is proposed to flexibly configure the suitable algorithms under various optimization scenarios. Two kinds of fitness landscape analysis techniques improved based on specific problem characteristics construct the state spaces for hierarchical reinforcement learning. Among them, an adaptive classification based on dynamic ruggedness of information entropy is designed to discern the complexity of problems, which serves as the basis for decision-making actions in upper-layer space. Additionally, an online dispersion metric based on knowledge is further presented to distinguish the precise landscape features in lower-layer space. In light of the characteristics of the state spaces, the hierarchical action spaces composed of meta-heuristics with disparate exploration and exploitation are designed, and various action selection strategies are introduced. Taking into account the real-time environment and algorithm evolution behavior, dynamic reward mechanisms based on evolutionary success rate and population convergence rate are utilized to enhance search efficiency. The experimental results on the IEEE Congress on Evolutionary Computation (CEC) 2017, CEC 2014, and large-scale CEC 2013 test suites demonstrate that the proposed HRLHH exhibits superiority in terms of accuracy, stability, and convergence speed, and possesses strong generalization.