Beyond Mouse and Keyboard: Novel Interaction for Visualization

Established: June 21, 2010

Microsoft continues to innovate in the NUI space, continuously providing new modalities for interacting with our computing systems. We are leveraging our expertise in NUI and developing a radically new way for people to more naturally access information that helps them not only explore their data but also present insights through simple interactive sketches. We aim to remove some of the interaction friction present in traditional desktop interfaces to enable people to more effectively explore, understand, and communicate data.

 

TouchPivot

TouchPivot is a novel interface that assists visual data exploration on tablet devices. With novices in mind, TouchPivot supports data transformations, such as pivoting and filtering, with simple pen and touch interactions, and facilitates understanding of the transformations through tight coupling between a data table and visualization. We bring in WIMP interfaces to TouchPivot, leveraging their familiarity and accessibility to novices.

 

SketchInsight

We advance research efforts in combining the casual sketching approach of whiteboards with the machine’s computing power. SketchInsight is a system that applies the familiar and collaborative features of a whiteboard interface to the accurate data exploration capabilities of interactive visualizations. SketchInsight enables data analysis with more fluid interaction, allowing people to visually explore their data by drawing simple charts and directly manipulating them.

 

WordlePlus

We open new possibilities of Wordle, a visualization technique commonly used to summarize texts. WordlePlus is an interactive authoring tool that leverages natural interaction and animation. It supports direct manipulation on words with pen and touch interaction, enabling more flexible wordle creation. WordlePlus introduces new two-word multi-touch manipulation, such as concatenating and grouping two words, and provides pen interaction for adding and deleting a word. In addition, WordlePlus employs animation to amplify the strength of Wordle, allowing people to create more dynamic and engaging wordles.

 

SketchStory

To create a new, more engaging form of storytelling with data, we leverage and extend the narrative storytelling attributes of whiteboard animation with pen and touch interactions. SketchStory is a data-enabled digital whiteboard that facilitates the creation of personalized and expressive data charts quickly and easily. SketchStory recognizes a small set of sketch gestures for chart invocation, and automatically completes charts by synthesizing the visuals from the presenter-provided example icon and binding them to the underlying data. Furthermore, SketchStory allows the presenter to move and size the completed data charts with touch, and filter the underlying data to facilitate interactive exploration.

 

TouchWave

We bring multi-touch interactions to a specific visualization technique, stacked graphs. Stacked graphs are a visually appealing and popular method for presenting time series data, however, they come with associated problems—issues with legibility, difficulties with comparisons, and restrictions in scalability. TouchWave is a rethinking and extension of stacked graphs for multi-touch capable devices that provides a variety of flexible layout adjustments, interactive options for querying data values, and seamlessly switching between different visualizations. In addition to ameliorating the main issues of stacked graphs, TouchWave also integrates hierarchical data within stacked graphs. We demonstrate TouchWave capabilities with two datasets—a music listening history and movie box office revenues—and discuss the implications for weaning other visualizations off mouse and keyboard.

People

Publications

Videos