Improved Speech Reading through a Free-Parts Representation

Abstract

Motivated by the success of free-parts based representations in face recognition [1] we have attempted to address some of the problems associated with applying such a philosophy to the task of speaker-independent automatic speech read-ing. Hitherto, a major problem with canonical area-based approaches in automatic speech reading is the intrinsic lack of training observations due to the visual speech modality's low sample rate and large variability in appearance. We believe a free-parts representation can overcome many of these limitations due to its natural ability to generalize by producing many observations from a single mouth image, whilst still preserving the ability to discriminate between various visual-speech units. This approach additionally re-quires a modification to traditional techniques employed for the estimation of hidden Markov Models (HMMs), whose resultant models we currently refer to as free-parts HMMs (FP-HMMs). Results will be presented on the CUAVE audio-visual speech database.