A human-computer interaction system
This project aimed to create a robust, scalable, gesture based human-computer interaction system.
A camera and a projector are mounted on the ceiling and project an image down onto a screen constructed from a partially retro-reflective screen material. The screen is constructed of a laminate of polyester, sheer fabric, and a reflective microstructure film (provided by Reflexite Corporation).
The screen is partially retro-reflective, so the camera sees only the projected image. No additional lighting is required. This screen structure allowing self-calibration, and the screen does not need to be a fixed distance from the projector, nor does it need to be flat. Any object that intersects the projected image appears black – even white paper. Gestures can be detected by thresholding the camera image, then looking for closed surfaces with a blob detection algorithm. Looking at metrics like size, shape and neighboring objects lets us differentiate a hand from a business card from a laptop. Retro-reflective tags can even be used as controls or to identify mobile devices.
Examples of how the system can be used:
A blank sheet of paper is used as a magnifier on maps:
Pinch gestures are used to create multiple simultaneous virtual paintbrushes – where the aspect ratio of the gesture sets the width of the paintbrush:
And gestures are used as controls – shown here controling a virtual musical instrument: