Microsoft Research collaborates with KAIST in Korea to explore bimanual interactions with haptic feedback in virtual reality

Published May 6, 2021

By Michel Pahud , Principal Research Software Development Engineer Mike Sinclair , Senior Principal Researcher Andrea Bianchi , Associate Professor at KAIST

Share this page

Editor’s Note: Bimanual controllers are frequently used to enhance the realism and immersion of virtual reality experiences such as games and simulations. Researchers have typically relied on mechanical linkages between the controllers to recreate the sensation of holding different objects with both hands. However, those linkages cannot quickly adapt to simulate dynamic objects. They also make for bulky controllers that can’t be disconnected to support free, independent movements. This is the problem that researchers seek to solve in the recent paper titled “GamesBond: Bimanual Haptic Illusion of Physically Connected Objects for Immersive VR Using Grip Deformation”.

GamesBond is the outcome of a recent collaboration between Michel Pahud and Mike Sinclair from Microsoft Research and Andrea Bianchi, associate professor at KAIST and director of the MAKinteract (opens in new tab) lab, with two of his students, Neung Ryu, the original author of the paper, and Hye-Young Jo. The paper was accepted at ACM CHI Conference on Human Factors in Computing Systems (CHI 2021) (opens in new tab), where it received an honorable mention award.

In this project, we explored a pair of novel 4-DoF controllers, without actual physical linkage between them, that can bend, twist, and stretch together in concert to create the illusion of being connected as a single device with a physical link. Each controller can bend from 0 to 30 degrees in any direction, twist from -70 degrees to 70 degrees and stretch from -2.5mm to 9.0mm (the paper provides all the details of the mechanism).

Figure 1: The controller can move in multiple directions, including bending, twisting and stretching

Figure 2: The bonds can be (a) rigid to create an experience such as holding a solid bar with two hands, (b) soft to create deformable and pliable objects, or (c) dynamic and capable of rendering inertia and oscillations to perceive objects like elastic ropes.

Figure 3: The controllers open doors for many user experiences, such as jumping rope held by (a) one or (b) two people

Figure 4: The controllers can create a force illusion that feels like breaking a bar

Figure 5: The controllers can create the illusion of moving weight, such as when holding a fishing net

Figure 6: Applications can render different grip shapes, such as: (a) a pair of separate electric screwdrivers with independent handles, (b) a hammer drill with parallel handles, or (c) a jackhammer with aligned handles…

Figure 7: A fishing rod application can give the illusion of a) pulling a line and b) winding a winch.

These figures are just a few examples explored, but the sky is the limit.

We conducted two complementary technical evaluations to characterize the rendering capabilities of the GamesBond controllers – a pilot study with four participants and a user study with 12 participants. Both quantitative and qualitative metrics support the hypothesis that GamesBond increases the realism, immersion and enjoyment of a VR experience. It also shows that the device can best render soft and dynamic bodies (e.g., a jumping rope), indicating possible avenues of exploration for future applications.

How did this collaboration with KAIST begin

This collaboration began by chance when Pahud and Bianchi met at the ISS2019 (opens in new tab)conference in Daejeon, Korea, and had side discussions about collaboration on novel user experiences. Bianchi showed Pahud his lab and its recent projects. “I was really impressed about the environment of the lab, the amount of innovative human computer interaction (HCI) hardware projects going on there, and the quality of the students. I thought we definitely have to find a way to collaborate together,” said Pahud.

After staying in touch even despite the COVID-19 pandemic, Bianchi’s students at the MAKinteract lab showed Pahud an early prototype of a controller that can bend, twist and stretch. It was clear that we would need two of these controllers to create a bimanual interaction and the feel of a virtual link between them. We quickly decided to invite Mike Sinclair to join and share his expertise in novel hardware and advise the students how to improve the controllers.

Both Pahud and Sinclair, along with former colleague Hrvoje Benko, had previously worked together on innovative hardware and experiences such as the TouchMover haptic feedback display, a 3D touchscreen which won a best paper award at ISS2014.

In this video, Michel Pahud demonstrates the research project of an TouchMover: Actuated 3-D Display with Haptic Feedback. The system consists of a touchscreen with a robot arm, engineered for instant, sensitive responsiveness, smooth forward and backward movement, and applications that support multi-touch screen interactions, force sensing, 3-D visualizations and depth movement.

“This collaboration with KAIST has been amazing,” Pahud said, “Andrea and his students are so talented and able to very quickly build compelling and novel hardware. We met regularly and every time I suggested new experiences or modifications, they implemented it by the next meeting! It’s impressive to see how quickly they make progress, which allowed us to submit to the CHI conference in time. We keep meeting regularly now and I am truly looking forward to future research projects with Andrea’s team.” Pahud was also pleased to join the PhD committee for one of Bianchi’s PhD students.

To learn more about GamesBond, please read the paper or play the video demo. You can also attend the CHI session: GamesBond: Bimanual Haptic Illusion of Physically Connected Objects for Immersive VR Using Grip Deformation on Thursday, May 13 at 10:00 PST.