Over the course of the COVID-19 pandemic, some truths about the nature of work have been underscored: it is uniquely complex, quickly shifting, and increasingly technology-mediated. Teaching, medicine, mental health, and other professions—previously thought to be near-impossible to do remotely—have all abruptly moved to online and hybrid mediums. All kinds of workers have needed to find new and creative ways to do their jobs. For many, the boundary between office and home have become a thin, blurred line. These changes have not been small, and they have not been without growing pains.
- Project The New Future of Work
Situated in this landscape, researchers are now tasked with the challenge of trying to understand how these short-term transformations will impact workplaces and work cultures in the long term. Taking place a little over a year after many quick-pivot workplace changes unfolded, the 2021 ACM Virtual Conference on Human Factors in Computing Systems (CHI 2021) is a timely moment for researchers to present some of their initial findings about how workplaces and people have adapted this past year, how technology has helped to aid in these transitions, and where technology needs to grow to support a new future of work.
A look at a cross-section of research from Microsoft accepted and presented at CHI 2021 reveals that among these changes to the workplace, researchers are also thinking about the important role that frameworks for interpretability and privacy will play in the workplaces of the future—especially as they pertain to AI becoming a more effective tool for workplace insights and collaboration. Some CHI papers confront interpretability and trust head on, like “AI Trust Score: A User-Centered Approach to Building, Designing, and Measuring the Success of Intelligent Workplace Features” and “Manipulating and Measuring Model Interpretability,” while some have a narrower focus on how AI might impact certain fields, as in “Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens.”
About Microsoft Research
Advancing science and technology to benefit humanity
One concrete subject seems to be at the crossroads of the many elements mentioned above: remote meetings. Recent findings in the Work Trend Index show that remote meetings are both a growing pain point for companies and workgroups as well as an effective outlet for collaboration between close teams during a period of isolation. The difference between a good remote meeting and a bad one could be the difference between a breakthrough idea or a breakdown in communication, and the technologies that enable these meetings are relying on recent advances to improve meeting quality for teams and individuals. When cutting-edge technology is employed as an environment for and facilitator of these ubiquitous workplace encounters, it’s important that it becomes a tool for streamlining, rather than impeding, a core piece of daily work.
Below, explore three research papers that examine remote meetings and presentations from both human and technology perspectives. One common theme from research into work this past year is that tools need to support not only the work itself, but also the well-being and growth of the people doing the work, and that sentiment can be found throughout these papers. As the move to a more flexible model of work becomes more widespread, findings show that there is plenty of room for imaginative solutions to improve the remote and hybrid meetings of the future.
To learn more about the broader range of work presented by Microsoft, a Champion sponsor of this year’s CHI event, check out the Microsoft Research at CHI 2021 page.
Playing to an audience: Bridging the gap between online and in-person presentations
Ten-second take from the researchers: Current online presentations provide limited to no audience feedback, making it very difficult to establish rapport and effectively adapt the content of the presentation. Inspired by cinematographic efforts, we created AffectiveSpotlight, a research bot built for Microsoft Teams that analyzes facial responses and head gestures of audience members, and dynamically spotlights the most expressive ones.
Diving deeper: The work presented in “AffectiveSpotlight: Facilitating the Communication of Affective Responses from Audience Members during Online Presentations” leverages recent advances in affect sensing to capture and facilitate communication of relevant audience signals during online presentations. Using an exploratory survey of 175 respondents, the researchers assessed the most relevant audience responses. These included confusion, engagement, and head nods among others. The team then implemented AffectiveSpotlight, a research bot built for Microsoft Teams that analyzes facial responses and head gestures of audience members and dynamically spotlights the most expressive ones.
Novel angle: Researchers have explored a wide variety of methods to facilitate audience feedback to online presenters. However, the majority of the methods usually rely on audience self-reports which are then provided to presenters in the form of flying emojis or abstract visualizations. In contrast, this work analyzes the facial responses and head gestures of audience members in real-time and explores providing the information in the form of original video feeds, which are more familiar and pose fewer cognitive demands to the presenter as opposed to aggregated data visualizations. The researchers purposely decided to avoid labeling the inferred responses to empower presenters to make their own personal interpretations based on the context and their experience.
Key findings: When evaluating the experience of different presentation systems in a within-subjects study with 14 groups and 117 participants, the researchers observed several statistically significant differences. In particular, presenters using the AffectiveSpotlight rated to be more significantly aware of their audience, spoke for a longer period of time, and self-assessed the quality of their talks more similarly to the audience members, compared to two control conditions (randomly-selected spotlight and default platform UI).
In a user study conducted by the researchers, 83.11% of the participants reported missing relevant audience feedback when presenting online.
Insights from users: The researchers conducted a user survey, where people reported that they would like to see confusion and engagement states more than other cognitive states. Also, participants reported wanting to see head nods more than other audience behaviors. In addition, presenters were less interested in seeing negative states, such as sleepiness or sadness, as compared to more positive states, such as engagement or excitement. This is possibly because negative feedback may indicate lack of audience interest, which might induce performance anxiety associated with giving formal presentations. After using AffectiveSpotlight, both presenters and audience members reported improvements. One presenter said, “AffectiveSpotlight helped create a feedback loop with the audience,” while an audience member compared the effectiveness of AffectiveSpotlight to another method, “When I was not convinced about the cons [of a particular slide] in the first talk [using AffectiveSpotlight], I felt the presenter spent a lot of time trying to get their point unlike the second talk [using another method].” Check out the paper for more feedback from users and how the team conducted its surveys.
Potential future investigations for the research community: While the results of the study were very encouraging, it is important to note that the experiment was performed in controlled settings with a limited sample size. More research needs to be done in less controlled settings to better assess the generalization of the findings to more varied real-life online meetings. In addition, this work focused on the analysis of non-verbal cues to inform the spotlight selection, but future efforts may consider a multimodal approach that combines multiple sensing modalities (for example, microphones and eye tracking) to help improve the quality of the information provided.
The research team includes: Javier Hernandez (opens in new tab), Daniel McDuff (opens in new tab), Kael Rowan (opens in new tab), Jina Suh (opens in new tab), and Mary Czerwinski (opens in new tab) of Microsoft Research, along with Prasanth Murali, (opens in new tab) PhD Candidate at Northeastern University, who was a Research Intern with Microsoft Research at the time of the work.
MeetingCoach: Personalized post-meeting dashboard for contextual and behavioral insights
Ten-second take from the researchers: Meetings, and nowadays remote meetings, are an essential part of work, but many employees find them frustrating and even counter-productive when good meeting practices are lacking or violated. We developed an AI system to automatically create personalized post-meeting dashboards for each participant that captured contextually grounded behavioral dynamics. We found that reviewing the dashboard helped improve attendees’ awareness of meeting dynamics, with implications for improved effectiveness and inclusivity.
Diving deeper: The research in “MeetingCoach: An Intelligent Dashboard for Supporting Effective and Inclusive Meetings” involves human-centered design of a post-meeting dashboard to support improving the inclusivity and effectiveness of remote meetings. MeetingCoach is an intelligent post-meeting feedback dashboard that summarizes contextual and behavioral meeting information. Researchers performed an exploratory survey study of 120 information workers to identify important signals (turn taking, sentiment for example) and used these insights to wireframe prototypes. These designs were evaluated with 16 in situ participants who helped identify the components they would prefer in a post-meeting dashboard. After recording video-conferencing meetings of eight teams over four weeks, the team developed an AI system to quantify the desired meeting features and created personalized dashboards for each participant. Through interviews and surveys of 23 individuals, the researchers found that reviewing the dashboard helped improve attendees’ awareness of meeting dynamics, with implications for improved effectiveness and inclusivity.
Novel angle: This paper presents the first implementation of a multimodal (audio-visual) AI system for summarizing contextually grounded behavioral dynamics within remote meetings. In a human-centered manner, the researchers designed a dashboard that supports private reflection on meeting dynamics and shows potential for supporting more inclusive and effective meetings.
Key findings: Simply reviewing the AI-generated dashboard helped improve attendees’ awareness of meeting dynamics. It was easy to review where in the meeting there was consensus, dissent, or confusion. Also, people want meeting feedback at varying times. Sometimes presenting recommended actions and behavioral information immediately before the following meeting in a series would be most actionable for them—rather than directly after the previous meeting.
In a survey conducted by the researchers, 78% of the participants reported understanding social signals to be very important in meetings and 55% of the participants expressed that social signals are more difficult to understand in online meetings.
Insights from users: Many employees find meetings frustrating and even counter-productive, especially when best meeting practices are lacking or violated. The violations of general meeting norms and disrespectful behaviors have been shown to be negatively correlated with meeting effectiveness and satisfaction. The fractured nature of video conferencing can aggravate negative outcomes and marginalize some members of the meeting. Participants also provided recommendations for improving the dashboard. They suggested having fewer and/or customizable features and the need for actionable suggestions that could help them effectively modify their dynamics in the future. Many of these findings could be equally helpful for in-person or hybrid meetings as the goal is to support reflection on meeting dynamics after or in-between meetings.
Potential future investigations for the research community: A primary goal of remote collaboration tools should be to support the most effective meetings possible for all participants. The researchers argue that intelligent, reflective post-meeting dashboards can be useful to help promote positive meeting practices and help raise the awareness of participants about potentially harmful or counter-productive meeting dynamics. The researchers hope that those who seek to build similar meeting support tools adopt their design suggestions:
- Provide actionable suggestions both after and directly before a next meeting in a sequence
- Modify feedback design based on its delivery timing
- Provide training opportunities based on feedback from a series of meetings
- Incorporate multi-modal signals to provide rich feedback.
The research team includes: Daniel McDuff, Robert Sim, Jina Suh, Kael Rowan, Javier Hernandez, Sean Rintel, Kevin Moynihan, and Mary Czerwinski of Microsoft Research, along with Samiha Samrose, PhD Student at University of Rochester, who was a Research Intern with Microsoft Research at the time of the work.
Analyzing the prevalence and impact of multitasking behavior during remote meetings
Ten-second take from the researchers: Finding it hard to stay focused during remote meetings and trying to get other things done? You are not the only one! We conducted a large-scale analysis of remote meeting multitasking behavior with US Microsoft employees. A key finding was that the larger the meeting, the more likely participants will multitask, and that isn’t necessarily a bad thing.
Diving deeper: Remote meetings are critical for remote work because of the need for synchronous collaboration in the absence of in-person interactions. In the paper “Large Scale Analysis of Multitasking Behavior During Remote Meetings,” researchers found that in-meeting multitasking is closely related to people’s productivity and wellbeing. In this paper, researchers present what they believe is the most comprehensive study of remote meeting multitasking behavior, through an analysis of a large-scale telemetry dataset, collected from February to May 2020 on US Microsoft employees, and a 715-person diary study from mid-April to mid-August 2020. Findings suggest important best-practice guidelines for remote meetings, some of which include avoiding important meetings in the morning and setting expectations for attention. They also indicate design implications for productivity tools, like support for a scale of attendance attention level and positive remote multitasking.
Novel angle: Both good and bad types of multitasking exist, though we often focus on the negative aspects. The findings suggest it is important to encourage positive aspects of multitasking while reducing its negative implications. Not every meeting requires every participant’s full attention at all times, and productivity tools could help facilitate positive multitasking by:
- Supporting “focus mode” for remote meetings to eliminate distractions from software interfaces during meetings.
- Supporting other types of engagement during remote meetings (like note taking through better interface design, such as supporting split views for multitasking).
- Supporting scaled meeting engagement (from high to low) in invitations and interfaces, to help set attention expectations.
- Providing features for organizers to rank the importance the full meeting/certain aspects of the meeting to help attendees decide which meetings to attend and when to skip some parts of the meetings.
Key findings: The results demonstrate that intrinsic meeting characteristics such as size, length, time, and type significantly correlate with the extent to which people multitask, and multitasking can lead to both positive and negative outcomes. A few key takeaways include that multitasking happens more in morning meetings and that multitasking can sometimes help boost productivity.
According to the results of this research, around 30% of remote meetings include email multitasking, and around 25% include file multitasking, like working on Word documents.
Human insights: This research shows that both intrinsic meeting characteristics and external factors are correlated with how people multitask during remote meetings. External factors included needing to catch up on other work or anxiety relief among others. People multitask more in large, long, morning, and recurring meetings, as well as meetings with lower relevance/engagement. Also, people multitask for both work-related tasks and non-work-related tasks. Organizers of meetings might consider a few best practices this research puts forward: avoiding important meetings in the morning, reducing the number of unnecessary meetings, shortening meeting duration while adding healthy breaks, setting attention expectations that encourage active contribution and participation when valuable, and allowing space for positive multitasking. The researchers note that it’s important to consider specific worker and workplace contexts when applying these practices.
Potential future investigations for the research community: The researchers identified several areas for future exploration. Since this paper focused specifically on information workers in the US, it would be interesting to learn more about how different worker types and country’s work cultures impact these results. Another avenue may be to compare multitasking behavior in hybrid versus fully remote settings. Building collaboration tools with the design implications laid out in the paper and deeper analysis of whether certain multitasking behaviors were positive or negative also hold potential for further insights.
The research team includes: Shamsi Iqbal, Mary Czerwinski, Sean Rintel, Brent Hecht, Jaime Teevan, and Longqi Yang of Microsoft Research; Hancheng Cao, PhD Candidate at Stanford University, who was a Research Intern with Microsoft Research at the time of the work; Priscilla Wong, PhD Student at University College London, who was a Research Assistant with Microsoft Research at the time of the work; and Chia Jung-Lee, who was with Microsoft Research at the time of the work.
Homing in on how chat is used to multitask during remote meetings
“The Promise and Peril of Parallel Chat in Video Meetings for Work” finds that simultaneous text, images, files, and links posted in meeting chat areas are now an integral part of remote work meetings. While chat paralleling audio-video has been investigated in other contexts, this is the first study documenting chat in remote meetings for work. It turns out that many remote meetings only work as well as they do because parallel chat is present, and without it some meetings would fail to work at all. As with other multitasking during meetings, parallel chat has pros and cons. Parallel chat can enable more attendees to have a say while avoiding interrupting speakers, as well as a place for detail, affirmations, and even humor. However, it can also be distracting from the main purpose of the meeting and difficult to follow. The researchers suggest potential best practices for managing chat, and they provide design suggestions for improving the display and productive value of chat in products like Microsoft Teams.
The research team includes: Advait Sarkar, Sean Rintel, Rachel Bergmann, Sharon Gillett, Danielle Bragg, Nancy Baym, and Abigail Sellen with Microsoft Research; and Damian Borowiec, PhD Student at Lancaster University, who was a contractor with Microsoft Research at the time of the work.
Discover more from CHI 2021 and the New Future of Work
This is a sampling of work by Microsoft Researchers from CHI 2021 exploring the New Future of Work, and remote meetings are just one of many aspects of this future that researchers are investigating. If you are still working from home and are curious about how the transition back to the office might go, another CHI 2021 paper out of Microsoft Research Asia, titled “Returning to the Office During the COVID-19 Pandemic Recovery: Early Indicators from China,” explores how knowledge workers navigated the early phases of this transition and the key factors impacting these workers. To learn more about Microsoft researchers’ broader work toward making remote work more effective, check out this collection of resources. For an in-depth analysis of how people’s work lives have changed during the COVID-19 pandemic, you can read the report “New Future of Work: Research from Microsoft into the Pandemic’s Impact on Work Practices.”
Artificial intelligence 
Human-computer interaction