Multistage Temporal Difference Learning for 2048-Like Games

Szubert and Jaśkowski successfully used temporal difference (TD) learning together with n -tuple networks for playing the game 2048. However, we observed a phenomenon that the programs based on TD learning still hardly reach large tiles. In this paper, we propose multistage TD (MS-TD) learning, a k...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE transactions on computational intelligence and AI in games. Jg. 9; H. 4; S. 369 - 380
Hauptverfasser: Yeh, Kun-Hao, Wu, I-Chen, Hsueh, Chu-Hsuan, Chang, Chia-Chuan, Liang, Chao-Chin, Chiang, Han
Format: Journal Article
Sprache:Englisch
Veröffentlicht: IEEE 01.12.2017
Schlagworte:
2048
expectimax
Games
Law
Learning (artificial intelligence)
Learning systems
Measurement
stochastic puzzle game
temporal difference (TD) learning
threes
ISSN:1943-068X, 1943-0698
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Szubert and Jaśkowski successfully used temporal difference (TD) learning together with n -tuple networks for playing the game 2048. However, we observed a phenomenon that the programs based on TD learning still hardly reach large tiles. In this paper, we propose multistage TD (MS-TD) learning, a kind of hierarchical reinforcement learning method, to effectively improve the performance for the rates of reaching large tiles, which are good metrics to analyze the strength of 2048 programs. Our experiments showed significant improvements over the one without using MS-TD learning. Namely, using 3-ply expectimax search, the program with MS-TD learning reached 32768-tiles with a rate of 18.31%, while the one with TD learning did not reach any. After further tuned, our 2048 program reached 32768-tiles with a rate of 31.75% in 10,000 games, and one among these games even reached a 65536-tiles, which is the first ever reaching a 65536-tiles to our knowledge. In addition, MS-TD learning method can be easily applied to other 2048-like games, such as Threes. Based on MS-TD learning, our experiments for Threes also demonstrated similar performance improvement, where the program with MS-TD learning reached 6144-tiles with a rate of 7.83%, while the one with TD learning only reached 0.45%.
ISSN:1943-068X
1943-0698
DOI:10.1109/TCIAIG.2016.2593710