Risk-sensitive reinforcement learning

We derive a family of risk-sensitive reinforcement learning methods for agents, who face sequential decision-making tasks in uncertain environments. By applying a utility function to the temporal difference (TD) error, nonlinear transformations are effectively applied not only to the received reward...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Neural computation Ročník 26; číslo 7; s. 1298
Hlavní autoři: Shen, Yun, Tobia, Michael J, Sommer, Tobias, Obermayer, Klaus
Médium: Journal Article
Jazyk:angličtina
Vydáno: United States 01.07.2014
Témata:
Algorithms
Brain - physiology
Brain Mapping
Decision Making - physiology
Humans
Magnetic Resonance Imaging
Markov Chains
Models, Psychological
Nonlinear Dynamics
Oxygen - blood
Probability
Reinforcement (Psychology)
Risk
ISSN:1530-888X, 1530-888X
On-line přístup:Zjistit podrobnosti o přístupu
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:We derive a family of risk-sensitive reinforcement learning methods for agents, who face sequential decision-making tasks in uncertain environments. By applying a utility function to the temporal difference (TD) error, nonlinear transformations are effectively applied not only to the received rewards but also to the true transition probabilities of the underlying Markov decision process. When appropriate utility functions are chosen, the agents' behaviors express key features of human behavior as predicted by prospect theory (Kahneman & Tversky, 1979 ), for example, different risk preferences for gains and losses, as well as the shape of subjective probability curves. We derive a risk-sensitive Q-learning algorithm, which is necessary for modeling human behavior when transition probabilities are unknown, and prove its convergence. As a proof of principle for the applicability of the new framework, we apply it to quantify human behavior in a sequential investment task. We find that the risk-sensitive variant provides a significantly better fit to the behavioral data and that it leads to an interpretation of the subject's responses that is indeed consistent with prospect theory. The analysis of simultaneously measured fMRI signals shows a significant correlation of the risk-sensitive TD error with BOLD signal change in the ventral striatum. In addition we find a significant correlation of the risk-sensitive Q-values with neural activity in the striatum, cingulate cortex, and insula that is not present if standard Q-values are used.
Bibliografie:content type line 23
SourceType-Scholarly Journals-1
ObjectType-Correspondence-1
ISSN:1530-888X
1530-888X
DOI:10.1162/NECO_a_00600