Always-Available Mobile Interfaces

We have continually evolved computing to not only be more efficient, but also more accessible, more of the time (and place), and to more people. We have progressed from batch computing with punch cards, to interactive command line systems, to mouse-based graphical user interfaces, and more recently to mobile computing. Each of these paradigm shifts has drastically changed the way we use technology for work and life, often in unpredictable and profound ways.

With the latest move to mobile computing, we now carry devices with significant computational power and capabilities on our bodies. However, their small size typically leads to limited interaction space (diminutive screens, buttons, and jog wheels) and consequently diminishes their usability and functionality. This presents a challenge and an opportunity for developing interaction modalities that will open the door for novel uses of computing.

Our work addresses these challenges by appropriating both the human body and the surrounding environment as interaction canvases. We achieve this by leveraging sensors used in medical contexts, and by applying signal processing and machine learning techniques that extract data about gesture and human behavior from those sensors.

Projects

	Workout: Automatic Exercise Analysis Although numerous devices exist to track and share exercise routines based on running and walking, these devices offer limited functionality for strength-training exercises. We introduce a system for automatically tracking repetitive exercises — such as weight training and calisthenics — via an arm-worn inertial sensor, with no user-specific training and no intervention during a workout.
	AirWave: Non-Contact Haptic Feedback Using Air Vortex Rings We explore the use of air vortex rings to enable at-a-distance haptics. Unlike standard jets of air, which are turbulent and dissipate quickly, vortex rings can be focused to travel several meters and impart perceptible feedback.
	Phoneprioception: Enabling Mobile Phones to Infer Where They Are Kept We collected data from 693 participants to understand where people keep their phone in different contexts and why. Using this data, we identified three placement personas: Single Place Pat, Consistent Casey, and All-over Alex. We also built prototypes employing capacitive, multispectral, and accelerometer sensing to infer phone placements automatically.
	SoundWave: Using the Doppler Effect to Sense Gestures We present SoundWave, a technique that leverages commodity speakers and microphones to sense in-air gestures. We generate an inaudible tone, which gets frequency-shifted when it reflects off of moving objects; we measure this shift with the microphone to infer various gestures.
	HumAntenna: Sensing Gestures Using the Body as an Antenna Home environments frequently offer a signal that is unique to locations and objects within the home: electromagnetic noise. In this work, we use the body as a receiving antenna and leverage this noise for gestural interaction.
	PocketTouch PocketTouch utilizes capacitive sensing to detect finger-strokes through fabric (e.g., while your phone is still in your pocket).
	Muscle-Computer Interfaces Muscle-computer interfaces directly sense and decode human muscular activity rather than relying on physical actuation or perceptible user actions. We believe that this is the first step towards tapping into the vast amount of information contained within the human physiology.
	Skinput: Bioacoustic Sensing for Input We present Skinput, a technology that appropriates the human body for acoustic transmission, allowing the skin to be used as an input surface. In particular, we resolve the location of finger taps on the arm and hand by analyzing mechanical vibrations that propagate through the body.