ReconViguRation: Reconfiguring Physical Keyboards in Virtual Reality

  • Daniel Schneider
  • Alexander Otte
  • Travis Gesslein
  • Philipp Gagel
  • Bastian Kuth
  • Mohamad Shahm Damlakhi
  • Oliver Dietz
  • Per Ola Kristensson
  • Jörg Müller
  • Jens Grubert

IEEE Transactions on Visualization and Computer Graphics | , Vol 25(11): pp. 3190-3201

Author's Version | Related File

Best Paper of ISMAR 2019, Beijing China

Physical keyboards are common peripherals for personal computers and are efficient standard text entry devices. Recent research has investigated how physical keyboards can be used in immersive head-mounted display-based Virtual Reality (VR). So far, the physical layout of keyboards has typically been transplanted into VR for replicating typing experiences in a standard desktop environment. In this paper, we explore how to fully leverage the immersiveness of VR to change the input and output characteristics of physical keyboard interaction within a VR environment. This allows individual physical keys to be reconfigured to the same or different actions and visual output to be distributed in various ways across the VR representation of the keyboard. We explore a set of input and output mappings for reconfiguring the virtual presentation of physical keyboards and probe the resulting design space by specifically designing, implementing and evaluating nine VR-relevant applications: emojis, languages and special characters, application shortcuts, virtual text processing macros, a window manager, a photo browser, a whack-a-mole game, secure password entry and a virtual touch bar. We investigate the feasibility of the applications in a user study with 20 participants and find that, among other things, they are usable in VR. We discuss the limitations and possibilities of remapping the input and output characteristics of physical keyboards in VR based on empirical findings and analysis and suggest future research directions in this area.

Fig. 2. Input-output dimensions of reconfiguring physical keyboards in virtual reality with mapped example applications. The x-axis shows input mappings and the y-axis shows output mappings. FL: Foreign languages; EM: Emojis; SC: Special characters; PW: Secure password entry; BS: Browser shortcuts; WM = Word macros.

ReconViguRation: Reconfiguring Physical Keyboards in Virtual Reality. IEEE ISMAR 2019.

ISMAR 2019 Best Paper Daniel Schneider, Alexander Otte, Travis Gesslein, Philipp Gagel, Bastian Kuth, Mohamad Shahm Damlakhi, Oliver Dietz, Eyal Ofek, Michel Pahud, Per Ola Kristensson, Jörg Müller & Jens Grubert. ReconViguRation: Reconfiguring Physical Keyboards in Virtual Reality. In IEEE Transactions of Visualization and Computer Graphics (TVCG), 2019. Preprint:

Enhancing mobile work and productivity with virtual reality

As people work from home, new opportunities and challenges arise around mobile office work. On one hand, people may have flexible work hours and may not need to deal with traffic or long commutes. On the other hand, they may need to work at makeshift spaces, with less-than-optimal working conditions while physically separated from co-workers.  Virtual reality (VR) has the potential to change the way we work, whether from home or at the office, and help address some of these new challenges. We envision the future office worker to be able to work productively everywhere, solely using portable standard input devices and immersive head-mounted displays. VR has the potential to enable this by allowing users to create working environments of their choice and by relieving them of physical limitations, such as constrained space or noisy environments.  In this webinar, Microsoft Researcher Eyal Ofek presents a summary of research investigating opportunities and challenges for realizing a mobile VR office environment. In particular, you’ll learn how VR can be mixed with standard off-the-shelf equipment (such as tablets, laptops, or desktops) to enable effective, efficient, and ergonomic mobile knowledge work.  Together, we’ll explore:  The unique needs of developing immersive applications for productivity.  Ways in which non-physical virtual workspaces can make work more efficient.  How to adapt virtual workspaces to the physical environment.  Resource list: Towards a Practical Virtual Office for Mobile Knowledge Workers (Publication)  Pen-based Interaction with Spreadsheets in Mobile Virtual Reality (Publication) Haptic Pivot: On-demand handhelds in VR (Video)  ReconViguRation: Reconfiguring Physical Keyboards in Virtual Reality (Publication)  Haptic Controllers (Project page)  Eyal Ofek (Researcher profile)  *This on-demand webinar features a previously recorded Q&A session and open captioning.  Explore more Microsoft Research webinars: