Temporally Abstract Model-Based Reinforcement Learning with Variable-Length Skills

Abstract

Building agents that can reason and plan at various levels of temporal abstraction has long been a goal of artificial intelligence research. In the context of reinforcement learning (RL), performing temporal abstraction typically implies the use of behavioral primitives that are executed over extended periods of time. These behavioral primitives are sometimes known as options, but following more recent literature, we refer to them as skills. Many existing model-free and all model-based methods for temporal abstraction in RL use skills that execute for a fixed length of time, constraining an agent to a single level of temporal abstraction throughout the entirety of a task. Variable-length skills allow an agent to flexibly change its level of abstraction, which is desirable as many interesting tasks are composed of sub-tasks that can take vastly different amounts of time. Furthermore, state-dependent termination conditions allow a skill to end exactly once its sub-task has been completed and not simply after a predetermined amount of time.

This thesis presents a novel approach to temporal abstraction in model-based RL that allows agents to learn a set of variable-length skills, state-dependent termination conditions, and dynamics on the temporally coarsened timescale induced by those skills. We demonstrate in long-horizon tasks from Datasets for Deep Data-Driven Reinforcement Learning (D4RL) that variable-length skills induce more repeatable state transitions, enabling higher downstream task performance compared to fixed-length skills. Furthermore, our approach is competitive with existing offline RL algorithms while naturally enabling knowledge transfer to novel goals.