Experience from a Reading Tutor that listens: Evaluation purposes, excuses, and methods

Abstract

This chapter gives three good reasons to evaluate reading software, identifies three methods for doing so, and refutes three excuses for not evaluating - namely, that evaluation is premature, unnecessary, or will be done by others: (1) Wizard of Oz experiments help test whether (and clarify how) a proposed approach might work, and refute the excuse that evaluation is premature because the approach has not yet been implemented in a proposed system that may take years to develop. (2) Conventional controlled studies help determine whether an implemented system helps children gain more in reading than they would otherwise. This criterion is necessary to improve on the status quo, but the difficulty of meeting it refutes the excuse that evaluation is unnecessary due to the supposedly innate superiority of learning on computers, or of a proposed way to use them. (3) Experiments embedded in an automated tutor help analyze which tutorial actions help which students and words, thereby guiding improvement of the tutor in ways that third party evaluation cannot, thus refuting the excuse that evaluation can be left to others. The chapter details some practical lessons learned from designing, performing, and analyzing experiments embedded in Project LISTEN's school-deployed Reading Tutor, which uses speech recognition to listen to children read aloud, and is helping hundreds of children learn to read.