Efficient abstraction selection in reinforcement learning

Efficient abstraction selection in reinforcement learning

van Seijen, H.H.
Whiteson, S.
Kester, L.J.H.M.

2014

This article addresses reinforcement learning problems based on factored Markov decision processes (MDPs) in which the agent must choose among a set of candidate abstractions, each build up from a different combination of state components. We present and evaluate a new approach that can perform effective abstraction selection that is more resource-efficient and/or more general than existing approaches. The core of the approach is to make selection of an abstraction part of the learning agent's decision-making process by augmenting the agent's action space with internal actions that select the abstraction it uses. We prove that under certain conditions this approach results in a derived MDP whose solution yields both the optimal abstraction for the original MDP and the optimal policy under that abstraction. We examine our approach in three domains of increasing complexity: contextual bandit problems, episodic MDPs, and general MDPs with context-specific structure.

Physics & Electronics
DSS - Distributed Sensor Systems
TS - Technical Sciences
Infostructures
Information Society
Abstraction selection
Model-free learning
Reinforcement learning
Structure learning
Abstracting
Markov processes
Context-specific structures
Contextual bandits
Decision making process
Model-free learning
Resource-efficient

http://resolver.tudelft.nl/uuid:7482d37d-699b-4191-8971-b156313a4c86

https://doi.org/10.1111/coin.12016

520172

0824-7935

Computational Intelligence, 30 (4), 657-699

article