Microsoft Research collaborates with KAIST in Korea to explore bimanual interactions with haptic feedback in virtual reality


By , Principal Research Software Development Engineer , Senior Principal Researcher , Associate Professor at KAIST

Editor’s Note: Bimanual controllers are frequently used to enhance the realism and immersion of virtual reality experiences such as games and simulations. Researchers have typically relied on mechanical linkages between the controllers to recreate the sensation of holding different objects with both hands. However, those linkages cannot quickly adapt to simulate dynamic objects. They also make for bulky controllers that can’t be disconnected to support free, independent movements. This is the problem that researchers seek to solve in the recent paper titled “GamesBond: Bimanual Haptic Illusion of Physically Connected Objects for Immersive VR Using Grip Deformation”.

GamesBond is the outcome of a recent collaboration between Michel Pahud and Mike Sinclair from Microsoft Research and Andrea Bianchi, associate professor at KAIST and director of the MAKinteract lab, with two of his students, Neung Ryu, the original author of the paper, and Hye-Young Jo. The paper was accepted at ACM CHI Conference on Human Factors in Computing Systems (CHI 2021), where it received an honorable mention award.

In this project, we explored a pair of novel 4-DoF controllers, without actual physical linkage between them, that can bend, twist, and stretch together in concert to create the illusion of being connected as a single device with a physical link. Each controller can bend from 0 to 30 degrees in any direction, twist from -70 degrees to 70 degrees and stretch from -2.5mm to 9.0mm (the paper provides all the details of the mechanism).

These figures are just a few examples explored, but the sky is the limit.

We conducted two complementary technical evaluations to characterize the rendering capabilities of the GamesBond controllers – a pilot study with four participants and a user study with 12 participants. Both quantitative and qualitative metrics support the hypothesis that GamesBond increases the realism, immersion and enjoyment of a VR experience. It also shows that the device can best render soft and dynamic bodies (e.g., a jumping rope), indicating possible avenues of exploration for future applications.

How did this collaboration with KAIST begin

This collaboration began by chance when Pahud and Bianchi met at the ISS2019 conference in Daejeon, Korea, and had side discussions about collaboration on novel user experiences. Bianchi showed Pahud his lab and its recent projects. “I was really impressed about the environment of the lab, the amount of innovative human computer interaction (HCI) hardware projects going on there, and the quality of the students. I thought we definitely have to find a way to collaborate together,” said Pahud.

Spotlight: Webinar series

Microsoft research webinars

Lectures from Microsoft researchers with live Q&A and on-demand viewing.

After staying in touch even despite the COVID-19 pandemic, Bianchi’s students at the MAKinteract lab showed Pahud an early prototype of a controller that can bend, twist and stretch. It was clear that we would need two of these controllers to create a bimanual interaction and the feel of a virtual link between them. We quickly decided to invite Mike Sinclair to join and share his expertise in novel hardware and advise the students how to improve the controllers.

Both Pahud and Sinclair, along with former colleague Hrvoje Benko, had previously worked together on innovative hardware and experiences such as the TouchMover haptic feedback display, a 3D touchscreen which won a best paper award at ISS2014.  

In this video, Michel Pahud demonstrates the research project of an TouchMover: Actuated 3-D Display with Haptic Feedback. The system consists of a touchscreen with a robot arm, engineered for instant, sensitive responsiveness, smooth forward and backward movement, and applications that support multi-touch screen interactions, force sensing, 3-D visualizations and depth movement.

“This collaboration with KAIST has been amazing,” Pahud said, “Andrea and his students are so talented and able to very quickly build compelling and novel hardware. We met regularly and every time I suggested new experiences or modifications, they implemented it by the next meeting! It’s impressive to see how quickly they make progress, which allowed us to submit to the CHI conference in time. We keep meeting regularly now and I am truly looking forward to future research projects with Andrea’s team.” Pahud was also pleased to join the PhD committee for one of Bianchi’s PhD students.

To learn more about GamesBond, please read the paper or play the video demo. You can also attend the CHI session: GamesBond: Bimanual Haptic Illusion of Physically Connected Objects for Immersive VR Using Grip Deformation on Thursday, May 13 at 10:00 PST.