Kernel Conjugate Gradient for Fast Kernel Machines - Robotics Institute Carnegie Mellon University

Kernel Conjugate Gradient for Fast Kernel Machines

Conference Paper, Proceedings of 20th International Joint Conference on Artificial Intelligence (IJCAI '07), pp. 1017 - 1022, 2007

Abstract

We propose a novel variant of the conjugate gradient algorithm,Kernel Conjugate Gradient (KCG), designed to speed up learning for kernel machines with differentiable loss functions. This approach leads to a better conditioned optimization problem during learning. We establish an upper bound on the number of iterations for KCG that indicates it should require less than the square root of the number of iterations that standard conjugate gradient requires. In practice, for various differentiable kernel learning problems, we find KCG consistently, and significantly, outperforms existing techniques. The algorithm is simple to implement, requires no more computation per iteration than standard approaches, and is well motivated by Reproducing Kernel Hilbert Space (RKHS) theory. We further show that data-structure techniques recently used to speed up kernel machine approaches are well matched to the algorithm by reducing the dominant costs of training: function evaluation and RKHS inner product computation.

BibTeX

@conference{Ratliff-2007-9640,
author = {Nathan Ratliff and J. Andrew (Drew) Bagnell},
title = {Kernel Conjugate Gradient for Fast Kernel Machines},
booktitle = {Proceedings of 20th International Joint Conference on Artificial Intelligence (IJCAI '07)},
year = {2007},
month = {January},
pages = {1017 - 1022},
keywords = {Kernel Machine, Functional Gradient, Kernel Gradient, Conjugate Gradient, KD-tree, Gaussian Processes, Kernel Logistic Regression},
}