Interacting Above and Beyond the Display

Advances in sensing technology are poised to spark the next shift in human-computer interaction, liberating users from the 2D plane of interaction currently supported by the mouse and touchscreen. New sensors can read users’ shape and motion as they move about in three dimensions. What signal-processing algorithms and interaction models are appropriate for this mode of interaction above and beyond the screen? How can we use the more detailed, nuanced information made available by new sensors to enable more expressive interfaces, going beyond what a mouse can do but preserving its familiar predictability? The five articles in this issue deal with these questions, covering the spectrum from specialized sensing hardware to high-level interaction models, across multiple physical scales and applications.