A Least-Squares Unified View of PCA, LDA, CCA and Spectral Graph Methods

Abstract

Over the last century Component Analysis (CA) methods such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), Canonical Correlation Analysis (CCA) and Spectral Clustering(SC) have been extensively used as a feature extraction step for modeling, classification, visualization and clustering. This technical report proposes a unified framework to formulate many CA methods as a least-squares estimation problem. We show how many CA methods correspond to a particular instance of a weighted kernel reduced rank regression (KRRR). The least-squares formulation allows better understanding of normalization factors, and provides an easier generalization of CA techniques. In particular, we derive the matrix expressions for weighted generalizations of PCA, LDA, SC and CCA (including kernel extensions), and show its effectiveness on synthetic and real problems. Finally, we suggest an efficient numerical method to solve WKRRR.