A theoretical and empirical analysis of expected sarsa
conference paper
This paper presents a theoretical and empirical analysis of Expected Sarsa, a variation on Sarsa, the classic onpolicy temporal-difference method for model-free reinforcement learning. Expected Sarsa exploits knowledge about stochasticity in the behavior policy to perform updates with lower variance. Doing so allows for higher learning rates and thus faster learning. In deterministic environments, Expected Sarsa's updates have zero variance, enabling a learning rate of 1. We prove that Expected Sarsa converges under the same conditions as Sarsa and formulate specific hypotheses about when Expected Sarsa will outperform Sarsa and Q-learning. Experiments in multiple domains confirm these hypotheses and demonstrate that Expected Sarsa has significant advantages over these more commonly used methods. © 2009 IEEE.
Topics
TNO Identifier
279985
Publisher
IEEE
Source title
2009 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL 2009, 30 March - 2 April 2009, Nashville, TN, USA
Place of publication
Piscataway, NJ
Pages
177-184
Files
To receive the publication files, please send an e-mail request to TNO Repository.