The situated interaction research effort aims to enable computers to reason more deeply about their surroundings, and engage in fluid interaction with humans in physically situated settings. When people interact with each other, they engage in a rich, highly coordinated, mixed-initiative process, regulated through both verbal and non-verbal channels. In contrast, while their perceptual abilities are improving, computers are still unaware of their physical surroundings and of the “physics” of human interaction. Current human-computer interaction…
I am a principal researcher at Microsoft Research. My work is focused on applying sensing techniques to enable new styles of human-computer interaction. Today that means multi-touch and gesture-based interfaces, display technologies, depth cameras and so-called “natural” interfaces. In 2002 I helped found the Surface Computing group at Microsoft. I manage the Natural Interaction Research group at Microsoft Research.
Before joining Microsoft, I obtained my BA at Cornell University, and MS and PhD at the MIT Media Laboratory. I grew up in a small town in Pennsylvania and now live in the Wallingford neighborhood of Seattle.
You can view videos of my work and collaborators’ work on my YouTube channel.
November 2015: I am the proud recipient of ACM ICMI’s Lasting Impact Award for my 2004 paper “TouchLight: an imaging touch screen and display for gesture-based interaction“.
At BUILD 2015 we introduced the RoomAlive Toolkit, an open source SDK that calibrates multiple Kinect and multple projectors to enable interactive projection mapping experiences like that of RoomAlive and IllumiRoom. Ben Lower and I introduce the Toolkit and describe some of its components in a Channel9 video available here. Source code is on GitHub here.
I’ve been active in the UIST, CHI, Tabletop (now Interactive Tabletops and Surfaces), ICMI, and Ubicomp academic communities, where I’ve have had the pleasure of serving on various program committees. I co-chaired IEEE Tabletop 2007, chaired ACM UIST 2009 and ACM SUI 2014. I tend to publish under the name “Andrew D. Wilson“.
Established: May 4, 2017
The user of Virtual Reality, may find himself in different virtual worlds, such as a spy game (middle) or a space simulator(right) , yet they all give him tangible feedback using the same physical geometry in the real world (left). We propose a class of passive haptics that we call Sparse Haptic Proxy: a set of geometric primitives that simulate touch feedback in elaborate virtual reality scenes. Unlike previous passive haptics that replicate the…
Established: March 2, 2016
Room2Room is a life-size telepresence system that leverages projected augmented reality to enable co-present interaction between two remote participants. We enable a face-to-face conversation by performing 3D capture of the local user with color + depth cameras and projecting their virtual copy into the remote space at life-size scale. This creates an illusion of the remote person’s presence in the local space, as well as a shared understanding of verbal and non-verbal cues (e.g., gaze).
Established: July 1, 2015
Publications Hrvoje Benko, Christian Holz, Mike Sinclair, and Eyal Ofek NormalTouch and TextureTouch: High-fidelity 3D Haptic Shape Rendering on Handheld Virtual Reality Controllers. In Proc. of ACM UIST 2016. PDF Video Mahdi Azmandian, Mark Hancock, Hrvoje Benko, Eyal Ofek, and Andrew Wilson. Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences. In Proc. of ACM CHI 2016.
Established: October 3, 2014
We explore grip and motion sensing to afford new techniques that leverage how users naturally manipulate tablet and stylus devices during pen-and-touch interaction. We can detect whether the user holds the pen in a writing grip or tucked between his fingers. We can distinguish bare-handed inputs, such as drag and pinch gestures, from touch gestures produced by the hand holding the pen, and we can sense which hand grips the tablet, and determine the screen's…
Established: October 3, 2014
Mano-a-Mano is a unique spatial augmented reality system that combines dynamic projection mapping, multiple perspective views and device-less interaction to support face-to-face, or dyadic, interaction with 3D virtual objects. Its main advantage over more traditional AR approaches is users are able to interact with 3D virtual objects and each other without cumbersome devices that obstruct face to face interaction. ACM Digital Download Dyadic projected spatial augmented reality
Established: October 1, 2014
The RoomAlive Toolkit is an open source SDK that enables developers to calibrate a network of multiple Kinect sensors and video projectors. The toolkit also provides a simple projection mapping sample that can be used as a basis to develop new immersive augmented reality experiences similar to those of the IllumiRoom and RoomAlive research projects. The RoomAlive Toolkit is provided as open source under the MIT License. The code is available for download at GithHub: https://github.com/Kinect/RoomAliveToolkit.…
RoomAlive is a proof-of-concept prototype that transforms any room into an immersive, augmented, magical entertainment experience. RoomAlive presents a unified, scalable approach for interactive projection mapping that dynamically adapts content to any room. Users can touch, shoot, stomp, dodge and steer projected content that seamlessly co-exists with their existing physical environment.
Established: January 4, 2013
IllumiRoom is a proof-of-concept system from Microsoft Research. It augments the area surrounding a television screen with projected visualizations to enhance the traditional living room entertainment experience. Read the CHI 2013 Best-Paper: IllumiRoom: Peripheral Projected Illusions for Interactive Experiences (7.6 MB .pdf) IllumiRoom uses a Kinect for Windows camera and a projector to blur the lines between on-screen content and the environment we live in allowing us to combine our virtual…
Established: February 24, 2012
HoloDesk is a novel interactive system combining an optical see through display and Kinect camera to create the illusion that users are directly interacting with 3D graphics. Project Overview: HoloDesk is an interactive system combining an optical see through display and Kinect camera to create the illusion that users are directly interacting with 3D graphics. A virtual image of a 3D scene is rendered through a half silvered mirror and spatially aligned with…
Established: October 2, 2010
LightSpace combines elements of surface computing and augmented reality research to create a highly interactive space where any surface, and even the space between surfaces, is fully interactive. Our concept transforms the ideas of surface computing into the new realm of spatial computing. Instrumented with multiple depth cameras and projectors, LightSpace is a small room installation designed to explore a variety of interactions and computational strategies related to interactive displays and the space that they…
FoveAR: Combining an Optically See-Through Near-Eye Display with Spatial Augmented Reality ProjectionsHrvoje Benko, Eyal Ofek, Feng Zheng, Andy Wilson, in UIST '15 Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, ACM, November 5, 2015,
Juggling the Effects of Latency: Motion Prediction Approaches to Reducing Latency in Dynamic Projector-Camera SystemsJarrod Knibbe, Hrvoje Benko, Andy Wilson, Microsoft Research, May 11, 2015,
CrossMotion: Fusing Device and Image Motion for User Identification, Tracking and Device AssociationAndy Wilson, Hrvoje Benko, in ICMI '14 Proceedings of the 16th International Conference on Multimodal Interaction, ACM, November 12, 2014,
Your Phone or Mine?: Fusing Body, Touch and Device Sensing for Multi-User Device-Display InteractionStewart Tansley, Andy Wilson, A.J. Brush, in CHI '12 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, May 5, 2012,
DepthTouch: Using Depth-Sensing Camera to Enable Freehand Interactions On and Above the Interactive SurfaceHrvoje Benko, Andy Wilson, Microsoft, March 4, 2009,
DocuDesk: An Interactive Surface for Creating and Rehydrating Many-to-Many Linkages among Paper and Digital DocumentsKatherine Everitt, Meredith Ringel Morris, A.J. Brush, Andy Wilson, in 3rd IEEE International Workshop on Horizontal Interactive Human Computer Systems, 2008. TABLETOP 2008., IEEE Computer Society, October 28, 2008,
BlueTable: Connecting Wireless Mobile Devices on Interactive Surfaces Using Vision-Based HandshakingAndy Wilson, Raman Sarin, in GI '07 Proceedings of Graphics Interface 2007, ACM, May 28, 2007,
October 11, 2016
May 12, 2016
October 31, 2014
October 5, 2014
October 5, 2014
October 5, 2014
October 1, 2014
January 29, 2013
January 9, 2013
January 9, 2013
April 16, 2012
April 16, 2012
March 6, 2012
RoomAlive Toolkit sample calibration files
The RoomAlive Toolkit calibrates multiple projectors and cameras to enable immersive, dynamic projection mapping experiences such as RoomAlive. This download includes sample calibration files which can be used with the RoomAlive Toolkit.