Beyond Mouse and Keyboard: Novel Interaction for Visualization

Established: June 21, 2010




Microsoft continues to innovate in the NUI space, continuously providing new modalities for interacting with our computing systems. We are leveraging our expertise in NUI and developing a radically new way for people to more naturally access information that helps them not only explore their data but also present insights through simple interactive sketches. We aim to remove some of the interaction friction present in traditional desktop interfaces to enable people to more effectively explore, understand, and communicate data.


Multimodal Interaction for Data Visualization

Multimodal interaction offers many potential benefits for data visualization. It can help people stay in the flow of their visual analysis and presentation, with the strengths of one interaction modality offsetting the weaknesses of others. Furthermore, multimodal interaction offers strong promise for leveraging data visualization on diverse display hardware including mobile, AR/VR, and large displays. However, prior research on visualization and interaction techniques has mostly explored a single input modality such as mouse, touch, pen, or more recently, natural language and speech. The unique challenges and opportunities of synergistic multimodal interaction for data visualization have yet to be investigated. With this workshop, we aim to build a community of researchers focusing on multimodal interaction, explore opportunities and challenges in our research, and establish an agenda for multimodal interaction research specifically for data visualization.



TouchPivot is a novel interface that assists visual data exploration on tablet devices. With novices in mind, TouchPivot supports data transformations, such as pivoting and filtering, with simple pen and touch interactions, and facilitates understanding of the transformations through tight coupling between a data table and visualization. We bring in WIMP interfaces to TouchPivot, leveraging their familiarity and accessibility to novices.



We advance research efforts in combining the casual sketching approach of whiteboards with the machine’s computing power. SketchInsight is a system that applies the familiar and collaborative features of a whiteboard interface to the accurate data exploration capabilities of interactive visualizations. SketchInsight enables data analysis with more fluid interaction, allowing people to visually explore their data by drawing simple charts and directly manipulating them.



We open new possibilities of Wordle, a visualization technique commonly used to summarize texts. WordlePlus is an interactive authoring tool that leverages natural interaction and animation. It supports direct manipulation on words with pen and touch interaction, enabling more flexible wordle creation. WordlePlus introduces new two-word multi-touch manipulation, such as concatenating and grouping two words, and provides pen interaction for adding and deleting a word. In addition, WordlePlus employs animation to amplify the strength of Wordle, allowing people to create more dynamic and engaging wordles.



To create a new, more engaging form of storytelling with data, we leverage and extend the narrative storytelling attributes of whiteboard animation with pen and touch interactions. SketchStory is a data-enabled digital whiteboard that facilitates the creation of personalized and expressive data charts quickly and easily. SketchStory recognizes a small set of sketch gestures for chart invocation, and automatically completes charts by synthesizing the visuals from the presenter-provided example icon and binding them to the underlying data. Furthermore, SketchStory allows the presenter to move and size the completed data charts with touch, and filter the underlying data to facilitate interactive exploration.



We bring multi-touch interactions to a specific visualization technique, stacked graphs. Stacked graphs are a visually appealing and popular method for presenting time series data, however, they come with associated problems—issues with legibility, difficulties with comparisons, and restrictions in scalability. TouchWave is a rethinking and extension of stacked graphs for multi-touch capable devices that provides a variety of flexible layout adjustments, interactive options for querying data values, and seamlessly switching between different visualizations. In addition to ameliorating the main issues of stacked graphs, TouchWave also integrates hierarchical data within stacked graphs. We demonstrate TouchWave capabilities with two datasets—a music listening history and movie box office revenues—and discuss the implications for weaning other visualizations off mouse and keyboard.