On the Estimation of α-Divergences

Abstract

We propose new nonparametric, consistent Renyi-α and Tsallis-α divergence estimators for continuous distributions. Given two independent and identically distributed samples, a “naive” approach would be to simply estimate the underlying densities and plug the estimated densities into the corresponding formulas. Our proposed estimators, in contrast, avoid density estimation completely, estimating the divergences directly using only simple k-nearest-neighbor statistics. We are nonetheless able to prove that the estimators are consistent under certain conditions. We also describe how to apply these estimators to mutual information and demonstrate their efficiency via numerical experiments.