Rapidly Scaling Dialog Systems with Interactive Learning

In personal assistant dialog systems, intent models are classifiers that identify the intent of a user utterance, such as to add a meeting to a calendar, or get the director of a stated movie. Rapidly adding intents is one of the main bottlenecks to scaling — adding functionality to — personal assistants. In this paper we show how interactive learning can be applied to the creation of statistical intent models. Interactive learning [10] combines model definition, labeling, model building, active learning, model evaluation, and feature engineering in a way that allows a domain expert — who need not be a machine learning expert — to build classifiers. We apply interactive learning to build a handful of intent models in three different domains. In controlled lab experiments, we show that intent detectors can be built using interactive learning, and then improved in a novel end-to-end visualization tool. We then applied this method to a publicly deployed personal assistant — Microsoft Cortana — where a non-machine learning expert built an intent model in just over two hours, yielding excellent performance in the commercial service.