Bisimulation Metrics for Continuous Markov Decision Processes

In recent years, various metrics have been developed for measuring the behavioral similarity of states in probabilistic transition systems [J. Desharnais et al., Proceedings of CONCUR'99, Springer-Verlag, London, 1999, pp. 258-273; F. van Breugel and J. Worrell, Proceedings of ICALP'01, Sp...

Full description

Saved in:
Bibliographic Details
Published in:SIAM journal on computing Vol. 40; no. 6; pp. 1662 - 1714
Main Authors: Ferns, Norm, Panangaden, Prakash, Precup, Doina
Format: Journal Article
Language:English
Published: Philadelphia, PA Society for Industrial and Applied Mathematics 01.01.2011
Subjects:
Algorithmics. Computability. Computer arithmetics
Algorithms
Applied mathematics
Applied sciences
Approximation
Computer science; control theory; systems
Decision making
Exact sciences and technology
Linear programming
Markov analysis
Miscellaneous
Theoretical computing
World
Markov process
Markov decision
Similarity
State space
Transition system
93E20
Substitution
continuous
60J05
Sampling
Approximation
Navigation
68T37
Estimation
Reinforcement learning
Linear programming
State space method
Algorithm
90C40
Bisimulation
Transition state
Markov decision process
Planning
metrics
Robot
Continuous system
ISSN:0097-5397, 1095-7111
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In recent years, various metrics have been developed for measuring the behavioral similarity of states in probabilistic transition systems [J. Desharnais et al., Proceedings of CONCUR'99, Springer-Verlag, London, 1999, pp. 258-273; F. van Breugel and J. Worrell, Proceedings of ICALP'01, Springer-Verlag, London, 2001, pp. 421-432]. In the context of finite Markov decision processes (MDPs), we have built on these metrics to provide a robust quantitative analogue of stochastic bisimulation [N. Ferns, P. Panangaden, and D. Precup, Proceedings of UAI-04, AUAI Press, Arlington, VA, 2004, pp. 162-169] and an efficient algorithm for its calculation [N. Ferns, P. Panangaden, and D. Precup, Proceedings of UAI-06, AUAI Press, Arlington, VA, 2006, pp. 174-181]. In this paper, we seek to properly extend these bisimulation metrics to MDPs with continuous state spaces. In particular, we provide the first distance-estimation scheme for metrics based on bisimulation for continuous probabilistic transition systems. Our work, based on statistical sampling and infinite dimensional linear programming, is a crucial first step in formally guiding real-world planning, where tasks are usually continuous and highly stochastic in nature, e.g., robot navigation, and often a substitution with a parametric model or crude finite approximation must be made. We show that the optimal value function associated with a discounted infinite-horizon planning task is continuous with respect to metric distances. Thus, our metrics allow one to reason about the quality of solution obtained by replacing one model with another. Alternatively, they may potentially be used directly for state aggregation. [PUBLICATION ABSTRACT]
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0097-5397
1095-7111
DOI:10.1137/10080484X