By John Roach
If you really want to learn to speak a foreign language, everyone knows that practice is essential. The challenge is to find someone with the time, patience and skill to help you practice pronunciation, vocabulary and grammar.
For people learning Chinese, Microsoft is aiming to fill that void with a new smartphone app that can act as an always available, artificially intelligent language learning assistant.
The free Microsoft Learn Chinese app is now available on Apple’s iOS platform.
The app aims to solve a problem that is familiar to any language learner who has spent countless hours in crowded classrooms listening to teachers, watching language-learning videos at home or flipping through stacks of flashcards to master vocabulary and grammar — only to feel woefully underprepared for real-world conversations with native speakers.
“You think you know Chinese, but if you meet a Chinese person and you want to speak Chinese, there is no way you can do it if you have not practiced,” explained Yan Xia, senior develop lead at Microsoft Research Asia in Beijing. “Our application addresses this issue by leveraging our speech and natural language processing technology.”
(left to right) Yan Xia, Jonathan Tien, Frank Soong
Teaching assistant
The application is akin to a teacher’s assistant, noted Frank Soong, principal researcher and research manager of the Beijing lab’s speech group, which developed the machine-learning models that power Learn Chinese as well as Xiaoying, a chatbot for learning English that the lab deployed in 2016 on the WeChat platform in China.
“Our application isn’t a replacement for good human teachers,” said Soong. “But it can assist by being available any time an individual has the desire or the time to practice.”
Human teachers, he noted, are limited in the time they can devote to coaching students on their speaking skills even in the best of circumstances such as private, one-on-one sessions. The application’s speech technology is robust enough to help students make progress toward their learning goals in the absence of a human partner.
The language learning technology relies a suite of AI tools such as deep neural networks that have been tuned by Soong’s group to recognize what the language learners are trying to say and evaluate the speakers’ pronunciation. They are based on a comparison with models trained on data from native speakers as well as the lab’s state-of-the art text-to-speech synthesis technology.
When individuals use the app, they get feedback in the form of scores, along with highlighted words that need improvement and links to sample audio to hear the proper pronunciation.
The app includes lessons for both beginners and intermediates. The lessons for beginners mainly help users to learn short words and sentences, and then provide lessons with read-after-me and dialogue to practice Chinese that users have just learned. The lessons for intermediates are the interactive lessons that help users learn how to navigate real-world scenarios such as renting an apartment and ordering food in a restaurant.
“The app will work with you as a language learning partner,” said Xia. “It will chat with you and give you feedback based on what you are saying.”
Language independent, culturally dependent
The Learn Chinese app was developed as part of an ongoing innovation incubation project at Microsoft’s Asia research lab in Beijing. The incubation project is one instance of a class of decades-long research within the lab on speech and language technologies, including recent breakthroughs in speech recognition, natural language understanding and text-to-speech synthesis.
The machine-learning models and neural networks that power Learn Chinese and Xiaoying are language independent – they work equally as well when helping people learning to speak Chinese or English. “If we are successful on one side, one direction, then we should be able to do that on the other side, the other direction,” said Soong.
The differences between the Chinese and English language-learning applications are cultural, noted Jonathan Tien, director of engineering in the Engineering Innovation Group at Microsoft Research Asia.
Tien’s team performs what’s known as deployment driven research – learning how the applications are used in the real world and tweaking models, interfaces and software architecture based on data collected from users. The process, he noted, is eye opening and rewarding.
“Chinese is our native language,” he said, “but learning Chinese is not native. It is difficult. During this process, we are learning a lot ourselves.”
The researchers are collaborating with human foreign language teachers to understand best practices and figure out how to deploy the language learning technology in a way that augments the teachers’ approach by encouraging frequent use and real-world language improvement.
“In today’s technology environment, you can deploy your application and improve it based on users’ feedback all the time,” said Tien. “That is the beauty of deployment driven research.”