Learning Families of Behaviors for Legged Locomotion using Model-Free Deep Reinforcement Learning

Abstract

Successful deployment of complex systems, such as articulated legged robots, require general solutions to planning and control because the high dimensional state spaces associated with such systems make it impractical to search for a new solution each time the robot encounters a different challenge. To achieve some level of generality, in this work, we present a framework for end to end learning of a set of parameterized families of behaviors that can be modulated by low dimensional sets of control parameters. We draw inspiration from Central Pattern Generators (CPGs), which use networks of oscillators, to form expressive low-dimensional parameterizations of locomotive behaviors. We do not directly use CPGs because their design requires significant domain knowledge and hand tuning. Instead, we turn to model-free deep reinforcement learning (RL) which offers a framework for learning behavioral policies by interacting with the environment, with minimal to no domain knowledge about the robot or its environment. The results from RL are still restrictive because they represent a single behavior whose characteristics cannot be easily modified. Therefore, this work presents a framework that brings together ideas from CPGs and model-free deep RL to enable expressive parameterizations of behaviors to be learned end-to-end by interaction with the environment.