Adam Troy had an idea. He was spending so much time searching through old emails and attachments to prepare for meetings: Could AI help him be better prepared for meetings? Could machine learning automatically identify the most relevant emails and documents he needed for his meetings? Troy, who works on a team that incubates projects that infuse AI into Office, built an early prototype application to see.
At the same time, Paul Bennett’s Information and Data Sciences team in Microsoft Research AI was in the second year of a project to help people find the right information, effortlessly. What information was not just relevant, but useful, as people go about their days?
When Troy demonstrated his prototype, Bennett recognized the opportunity to explore the usefulness of his group’s theories in the real world, and these two independent but aligned projects came together. Bennett recruited research intern Qian Zhao, whose expertise is in developing intelligent systems and measuring their usefulness, to develop a prototype feature as part of a research project aimed at understanding how people interact with email when preparing for meetings.
Spotlight: Event Series
Microsoft Research Forum
Join us for a continuous exchange of ideas about research in the era of general AI. Watch the first four episodes on demand.
After Zhao, Bennett, and Troy had proven out the core idea with research, Abhishek Arun, who works in productizing AI into Office, began building and implementing a feature that clicks on a Calendar event in Outlook and brings up related emails and documents that help the user prepare for that meeting.
This is one example of how a close partnership between research and product development can quickly turn good ideas and deep research understandings into product-ready features. Today, Arun’s team is testing the feature for Microsoft employees; if the research results scale to a broad set of customers, it may be applied in the future to commercially available versions of Outlook.
Introducing CAPERS
The feature is called a Calendar-Aware Proactive Email Recommender System (CAPERS). While the acronym is a mouthful, the concept is simple: Outlook understands meetings and their context well enough to recommend the key emails that might help someone plan for a meeting. Using AI, CAPERS proactively selects and recommends potentially useful emails to users based on their upcoming calendar events, with a particular focus on meeting preparation.
While this may sound like a trivial task, the amount of time spent in meetings and preparing for meetings is immense. According to a Brandeis University infographic, in the US, 11 million meetings are held every day and the average corporate employee spends 37 percent of their time in meetings. There’s nothing trivial about that!
Recommendation engines must accurately predict what people will want and need. AI can make the workflow of meeting preparation more productive, because it can predict which emails would actually help one prepare for a meeting. There is a relationship between content (email), location and time (calendar) that can be measured, and AI can identify those relationships with the highest predicted usefulness.
The team, with lead author Zhao, published its results in the proceedings of the 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. The paper covered the results of a survey, a task-based experiment, and a field experiment comparing multiple AI-based email recommenders.
Said Zhao, “Industry-wide, email and calendar recommendations are an under-explored area of AI research. There is a lot of research on algorithms, but less on practical efficiencies in workflows. That’s why this was an interesting research project.”
The results of the research showed that a large proportion of email access is related to meetings. The amount of email content, how recent the email is, the semantic similarity between the content describing the meeting and the content in the email, and the similarity between the people involved in the meeting and the people in the email thread all positively predict the usefulness of emails to meeting preparation, with the similarity in content being the most predictive. Basically, the recommender searches through emails and finds those it thinks might help one come better prepared for a meeting, because of topics, people, or calendar content.
Because of the privacy issues related to email and calendars, the machine learning models needed to work without much training data, and needed to find learning signals that were generalizable—the system had to keep the cloud data private. Four different machine learning models were tested and the results of the research showed that an online-learned linear model doubled user engagement when compared with the baselines.
Said Troy, “AI works best when it’s invisible. It shouldn’t announce itself as AI; it should just make your workflow better. CAPERS has given us the information to help create a feature that many people will find useful as they make their meeting preparation time more efficient.”
Recommendation technologies have been primarily developed for selling people products or finding better news and movies, but the underlying AI technology has unexplored potential to improve the ease of every person’s daily job. Said Bennett, “CAPERS demonstrates the rich research space available to incorporate AI directly at the point of interaction in productivity workflows. CAPERS also demonstrates how we work together at Microsoft to push the boundaries of science and the state-of-the-art in productivity software to build a better experience for every person.”
Artificial intelligence 
Data platforms and analytics 
Systems and networking