Multimodal Sensing for Explicit and Implicit Interaction

We present four perceptual user interface systems that explore a continuum from explicit to implicit interaction. Explicit interactions include most of today’s mouse and keyboard-based interaction models, where the user initiates a discrete action and expects a timely discrete response. Implicit interactions may use passive monitoring of the user over longer periods of time, and result in changing some aspect of the rest of the interaction. For example, less urgent notifications may be withheld from the user if the system detects they are engaged in a meeting. The first system is FlowMouse, a program that tries to emulate the mouse but suggests more implicit kinds of interaction. Second, we describe GWindows, which focuses complementing the mouse in today’s GUI in a way that might support casual interactions. Then, ToughtLight eschews the traditional notion of an explicit discrete pointer altogether and in so doing presents a number of challenges in designing applications. And finally, S-SEER supports a purely implicit style of interaction driven by models of situational awareness. In presenting this series of projects in order from explicit to implicit modes, we hope to illustrate by way of example the various challenges and opportunities for perceptual user interfaces.