Zoom Link

Abstract:
Multi-view machine learning has received substantial attention in various applications over recent years. These applications typically involve learning on data obtained from multiple sources of information, such as, for example, in multi-sensor systems such as self-driving cars and patient bed-side monitoring. Learning models for such applications can often benefit from leveraging not only the information from individual sources, but also the interactions and relationships between these sources.

In this proposal, we look at multi-view learning approaches which try to model these inter-view interactions explicitly. Here, we define interactions and relationships between views in terms of the information which is shared across these views, i.e. information redundancy between views. We distinguish between global relationships, which are shared across all views, and local relationships, which are only shared between a subset of views For example, in a multi-camera system, we can think of global relationships to be defined over the part of a scene which is visible to all cameras, while local relationships may exist between a subset of views to be defined by the intersection of the fields of view of only those cameras.

We consider three main aspects of modeling such inter-view relationships. First, we look at understanding relationships within multi-view data. We describe two methods which aim to uncover and model local relationships between views: (i) Robust Multi-view Auto-Encoder, which generalizes the idea of drop-out to views as a whole and (ii) One-vs-Rest Embedding Learning, which explicitly models the local relationships by considering each view separately. We also propose extensions to these methods, as well as alternate approaches to understanding inter-view relationships.

Next, we look at exploiting this understanding to solve down-stream tasks and real-world problems. Here, we use our proposed models to tackle real-world problems, and demonstrate the effectiveness of explicitly modeling inter-view relationships. We also discuss how we can extend our approaches to looking at special applications, such as dynamical systems and asynchronous multi-view data.

Finally, we discuss improving inter-view relationships by facilitating favorable interactions between views in multi-view data. We first show how we can re-interpret individual views as data points, allowing us to apply traditional machine learning approaches to modeling inter-view relationships. We then describe Scalable Active Search as a candidate approach for view-selection. We also propose additional methods to improve inter-view relationships using our view-as-data-point interpretation, and discuss ways for their online improvement.

More Information

Thesis Committee Members:
Artur Dubrawski, Chair
Jeff Schneider
Srinivasa Narasimhan
Junier Oliva, University of North Carolina, Chapel Hil