图 书 馆
非政策TD的固定点The Fixed Points of Off-Policy TD |
|
| 课程网址: | http://videolectures.net/nips2011_kolter_fixedpoints/ |
| 主讲教师: | J. Zico Kolter |
| 开课单位: | 卡内基梅隆大学 |
| 开课时间: | 2012-09-06 |
| 课程语种: | 英语 |
| 中文简介: |  |
| 课程简介: | Off-policy learning, the ability for an agent to learn about a policy other than the one it is following, is a key element of Reinforcement Learning, and in recent years there has been much work on developing Temporal Different (TD) algorithms that are guaranteed to converge under off-policy sampling. It has remained an open question, however, whether anything can be said a priori about the quality of the TD solution when off-policy sampling is employed with function approximation. In general the answer is no: for arbitrary off-policy sampling the error of the TD solution can be unboundedly large, even when the approximator can represent the true value function well. In this paper we propose a novel approach to address this problem: we show that by considering a certain convex subset of off-policy distributions we can indeed provide guarantees as to the solution quality similar to the on-policy case. Furthermore, we show that we can efficiently project on to this convex set using only samples generated from the system. The end result is a novel TD algorithm that has approximation guarantees even in the case of off-policy sampling and which empirically outperforms existing TD methods. |
| 关 键 词: | 计算机科学; 机器学习; 强化学习 |
| 课程来源: | 视频讲座网 |
| 最后编审: | 2020-04-09：cjy |
| 阅读次数: | 43 |