On the Convergence Rate of MCTS for the Optimal Value Estimation in Markov Decision Processes

A recent theoretical analysis of a Monte-Carlo tree search (MCTS) method properly modified from the "upper confidence bound applied to trees" (UCT) algorithm established a surprising result, due to a great deal of empirical successes reported from heuristic usage of UCT with relevant adjus...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE transactions on automatic control Ročník 70; číslo 7; s. 4788 - 4793
Hlavní autor: Chang, Hyeong Soo
Médium: Journal Article
Jazyk:angličtina
Vydáno: New York IEEE 01.07.2025
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Témata:
Approximation algorithms
Artificial intelligence
Complexity theory
Computational modeling
Convergence
Data mining
Estimation
Markov decision process (MDP)
Markov processes
Monte-Carlo tree search (MCTS)
multiarmed bandit (MAB)
Search problems
Training
Uncertainty
Upper bound
upper confidence bound 1 (UCB1)
upper confidence bound applied to trees (UCT)
ISSN:0018-9286, 1558-2523
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:A recent theoretical analysis of a Monte-Carlo tree search (MCTS) method properly modified from the "upper confidence bound applied to trees" (UCT) algorithm established a surprising result, due to a great deal of empirical successes reported from heuristic usage of UCT with relevant adjustments for various problem domains in the literature, that its rate of convergence of the expected absolute error to zero is <inline-formula><tex-math notation="LaTeX">O(1/\sqrt{n})</tex-math></inline-formula> in estimating the optimal value at an initial state in a finite-horizon Markov decision process (MDP), where <inline-formula><tex-math notation="LaTeX">n</tex-math></inline-formula> is the number of simulations. We strengthen this dispiriting slow convergence result by arguing within a simpler algorithmic framework in the perspective of MDP, apart from the usual MCTS description, that the simpler strategy, called "upper confidence bound 1" (UCB1) for multiarmed bandit problems, when employed as an instance of MCTS by setting UCB1's arm set to be the policy set of the underlying MDP, has an asymptotically faster convergence-rate of <inline-formula><tex-math notation="LaTeX">O(\ln n / n)</tex-math></inline-formula>. We also point out that the UCT-based MCTS in general has the time and space complexities that depend on the size of the state space in the worst case, which contradicts the original design spirit of MCTS. Unless heuristically used, UCT-based MCTS has yet to have theoretical supports for its applicabilities.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0018-9286
1558-2523
DOI:10.1109/TAC.2025.3538807