Abstract:
Optimal Control is a formulation for designing controllers for dynamical systems by posing it as an optimization problem, whereby the desired long-term behavior of the system is expressed using a cost function. The objective is to compute a policy, i.e. a mapping from the state of the system to its control inputs, that minimizes the specified cost function. A fundamental challenge in scaling up optimal policy synthesis for complex dynamical systems is that the computational requirement scales exponentially with the dimensionality of the state-space. Several approaches based on model reduction and hierarchical controls have been proposed to tackle this curse of dimensionality. Model reduction methods derive a lower-dimensional representation of the system to lessen the computational burden, whereas hierarchical approaches enforce a structure on the controller whereby policies for different inputs are only a function of some subsets of the entire state. However, most of these methods only reason about the open-loop dynamics of the system and are agnostic to the objective of the optimal control, and the reductions thus found may exhibit poor closed-loop behavior.

We introduce Policy Decomposition, a framework for hierarchical control, which unlike other approaches, finds simplifications to optimal control problems while accounting for their influence on the system’s closed-loop behavior. Policy Decomposition decouples and cascades the process of computing policies for different control inputs, whereby the policies are only a function of a subset of the state variables, leading to reduction in computation. For a given system, several such decompositions are valid, and Policy Decomposition assesses a priori how well the closed-loop behavior under the decomposition policy matches the one of the optimal policy. Additionally, we demonstrate the use of combinatorial search methods such as Genetic Algorithm and Monte Carlo Tree Search in efficiently finding decompositions that offer substantial reduction in computation while sacrificing minimally on optimality for a range of optimal control problems.

Hitherto, we have investigated optimal regulation problems for robotic systems of moderate complexity assuming noiseless actuation and access to an accurate model of their dynamics. We aim to devote the remainder of this thesis in extending the framework to tackle optimal trajectory tracking while accounting for uncertainty in the system dynamics. Furthermore, through empirical evaluations we propose to assess the utility of Policy Decomposition in simplifying highly complex optimal control problems.

Thesis Committee Members:
Hartmut Geyer, Chair
Christopher Atkeson
Zachary Manchester
Alex Gorodetsky, University of Michigan
Nikolai Matni, University of Pennsylvania

More Information