Efficient Handwriting Correction of Speech Recognition Errors with Template Constrained Posterior (TCP)

More mobile devices are starting to use automatic speech recognition for command or text input. However, correcting recognition errors in a small compact mobile device is usually inconvenient and it may take several finger operations on a small keypad to correct errors. In this paper, we propose a new multimodal input method and a novel confidence measure ― template constrained posterior (TCP) to simplify the correction process. The method works by interactively integrating a handwriting recognizer with a speech recognizer. Information obtained in pen-based error marking, like error location, error type, etc., is fed back to the speech recognizer, and speech recognition errors are automatically corrected using the TCP confidence measure. Experimental results on Aurora2, Wall Street Journal, Switchboard, and two Chinese databases show that compared with speech recognition baseline, the proposed method  achieves  relative error reduction of 64.9%, 43.9%, 26.1%, 39.0%, 31.4%, respectively, after the auto correction.