Microsoft Research Blog

Microsoft Research Blog

The Microsoft Research blog provides in-depth views and perspectives from our researchers, scientists and engineers, plus information about noteworthy events and conferences, scholarships, and fellowships designed for academic and scientific communities.

Project Zanzibar: Blurring the distinction between the digital and the physical worlds via tangible interaction in a portable implementation

April 9, 2018 | By Microsoft blog editor

It was a love of toys, a shared appreciation for the intrinsic beauty of physical objects and a recognition of their absence in the daily computer interactions of a world that currently spends most of its time gazing at and touching two-dimensional glass that propelled a team of Microsoft researchers in Cambridge, UK and Redmond, WA to embark on a quest to blur the divide between the digital and physical worlds.

The decidedly tangible fruit of their efforts is Project Zanzibar, a completely new sensing platform in the form of a flexible, portable mat that has the ability to locate, sense and communicate with objects as well as sense a user’s touch. The Project Zanzibar mat combines capacitive sensing and Near Field Communication (NFC) in a novel way, enabling multi-touch and hover gesture input to coexist with physical object manipulation and control. The Project Zanzibar research platform also introduces the power of portability in a tangible user interface because rather than provide its own display, it takes advantage of existing devices such as tablets. Roll it up, stow it and break it out at a picnic or on a train trip. Or in any room in the house that has a screen.

“The question driving us over a number of years was how you make interacting with a computer a very natural, invisible thing,” said Microsoft Senior Researcher Nicolas Villar. “Our fascination was with making the technology invisible so that you get the power of technology without it feeling like you are using technology.”

The scope of their achievements, detailed in a paper to be presented later this month at CHI 2018 in Montreal, is impressive. Supplementing the flexible form-factor Project Zanzibar mat, are hardware design patterns for tangible objects that build on NFC, leveraging and extending the capabilities of the mat.

“We began with a simple thought: what if we could blur the divide between the physical and digital worlds? What if you could play with toys and cards and blocks, while watching your actions come alive onscreen?” – Project Zanzibar Research Team

Any new research platform worthy of the name provides the ability to support a variety of compelling applications. The Project Zanzibar research team obviously intended not to disappoint anyone, developing a set of exciting scenarios in the play and learning spaces, two areas that the team is passionate about and in which the benefits of tangible interaction are well accepted. The team also validated the Project Zanzibar research platform by sharing deployments with professional game developers and incorporated the resulting experiences in creating physical/digital hybrid games.

With the portability and flexibility of the Project Zanzibar mat and its ability to leverage the ubiquity of equally portable screen technology, combined with the prevalence of homes with screens conceivably in every room, the team feels the moment for TUIs is at hand.

Scenarios

In their pursuit of a vision of humans interfacing with computers in which the focus is no longer on the computer or its peripherals, the Project Zanzibar team went the extra mile in imagining and prototyping delightful scenarios that convey a very compelling technological story.

Play

Every physical object in the Project Zanzibar world has a globally unique ID, enabling data to be stored for each unique object. This introduces the concept of a toy or object learning and possessing its own history and the capability to tell its story. Collectible cards can gain value over time. Dolls could conceivably “grow up” alongside a child, with shared experiences from playtime remembered and relatable (that is, replayable) over time.

Movie maker. Off-the-shelf toys were used to demonstrate the flexibility of Project Zanzibar for different types of play activities.

Project Zanzibar invites children to bring their toys into an interactive experience and watch them come alive through sound, visuals and special effects. The team had observed kids spending hours making stop-action movies to share online. They envisioned the Project Zanzibar movie maker scenario to let children tell stories by physically interacting with their toys on the Project Zanzibar mat while at the same generating an animated movie.

“The physical world is such a rich place, abundant with dimensionality and materiality. Our vision is to tap into that richness and imbue it with that digital magic and our hope is that Project Zanzibar is a step in that direction.” – Haiyan Zhang, Project Zanzibar Researcher

Kids can tell stories by manipulating toys and props on the mat to control corresponding graphical avatars, previewing and capturing the result as a movie on the screen of a connected device. Camera and light source objects frame and light the mise-en-scène. Additional props can be used to change the setting of a virtual scene. For example, a palm tree can change the virtual environment to a tropical setting. Movie maker is thus a story authoring and storytelling prototype, envisioned for children aged 5 to 12. In this age group, socio-dramatic or pretend play are prevalent play patterns, with children experimenting with role playing and ascribing unique personalities to their toys.

Toybox. Off-the-shelf toys were used to demonstrate the flexibility of Project Zanzibar for different types of play activities.

The toybox uses sound effects and audio dialogue to complement physical play. The toybox includes figurines for different characters (pirates and naval officers), the combination and proximity of different characters on the mat triggers lines of dialogue from a story — for example, two pirates will banter if placed close to one another, while a pirate will hail an officer with a friendly challenge. Through its globally unique ID, each pirate toy is unique, building on its narrative with each play session.

The sensing ability of the mat can be extended by the toy pieces, for example when the pirate climbs the tower, the mat detects when he reaches the top. Toy pieces can also become inputs; when the pirate fires his cannon, players hear the boom and later a splash when the cannonball lands in the sea.

Learning

The Project Zanzibar team exhibited an equal passion for imagining powerful opportunities for the research platform in the educational space, inspired by the Montessori Method. Developed over the late 19th and early 20th century, Maria Montessori’s theories of child development stressed the role of physical activity in absorbing academic concepts and practical skills. The Montessori method also emphasizes the importance of self-directed activity and adapting a child’s learning environment to his or her developmental level.

“The importance of multi-sensory learning is especially apparent in the Montessori method of teaching, where children use physical objects and self-directed activities for all subjects of learning. We took traditional Montessori exercises for young children and extended them with digital content and feedback using Project Zanzibar.” – the Project Zanzibar Research Team

The team took inspiration from traditional Montessori exercises for young children and extended them with digital content and feedback using Project Zanzibar. For example, the Letter Plates Montessori exercise provides feedback as words are formed and tactile letter-shapes are traced. Paper overlays on the Project Zanzibar mat do not interfere with functionality and provide additional context to the exercise.

Montessori-inspired letter plates with Project Zanzibar mat and tablet

The team envisions a rich family of educational apps including spelling and simple coding exercises using blocks, again with or without screens. By allowing kids to incorporate their tactile, natural world actions and senses when interfacing with computers, learning can conceivably be enhanced.

Implementation: Making the Tangible Real

The number of innovations and implementation decisions present in the Project Zanzibar mat and research platform defy encapsulation in a mere blog article and interested readers are encouraged to download the comprehensive technical paper by team members Villar, Zhang and their Microsoft Research colleagues Daniel Cletheroe, Greg Saul, Christian Holz, Tim Regan, Oscar Salandin, Misha Sra, Hui-Shyeong Yeo, and William Field.

Project Zanzibar incorporates scalable NFC coverage that can localize tag, capacitive sensing to detect object footprints, touch, and hover input and a tracking system that fuses NFC-based and capacitive-based tracking information to report interaction events. Manufacturing innovations led to a rollable mat surface. Hardware design includes patterns for tangible objects in order to track their rotation, harvest power, enable on-object input/output capabilities, extend sensing outside the bounds of the mat surface, and detect the presence and order of stacked objects.

Project Zanzibar Mat: Some Technical Detail

The Project Zanzibar mat prototype is designed for self-contained operation— all sensing and processing happens in the mat itself; high-level interaction events are sent to connected devices via USB or Bluetooth.

The mat incorporates scalable and localized NFC coverage and has the ability to energize and communicate with multiple tags up to 30mm above the surface of the mat. Innovations such as capacitive object footprints and touch and hover recognition detect gestures such as swiping across the mat or drawing shapes in the air.

Additional innovations solve problems in NFC and capacitive coexistence; for example, NFC communication and capacitive sensing must happen concurrently for the touch and object tracking system to work.

The tracking system fuses data from the NFC and capacitive subsystems to generate high-level interaction events.

Design Patterns for Project Zanzibar Tangibles

The Project Zanzibar team strove to ensure experiences that went beyond mere interaction with simple tokens, which offer only presence, location and identity as input. The researchers developed design patterns and hardware reference designs that simplify the creation of tangible objects with a range of flexibility across cost, complexity, and interactive capabilities. Simple NFC sticker tags were determined to support innovative design decisions across rotation, power harvesting, input/output, and extending and stacking.

One reads all this and senses that the moment for tangible user interfaces indeed is upon us. And if the passion of this research can be as inspiring in the outside world, then we may indeed be at a turning point in the realization of natural interactions with computers that lead to the next era of how we think of – or don’t think of! – computers when we are using them to create, learn, play and build.

Project Zanzibar has been a research exploration and the researchers are glad to share its publication with the wider HCI community. Microsoft Research pushes the boundaries of research in diverse areas, although not all projects may be pursued beyond the research prototype stage.

Up Next

Karin strauss podcast

Hardware and devices, Systems and networking

Storing digital data in synthetic DNA with Dr. Karin Strauss

Episode 48, October 31, 2018 - Dr. Strauss talks about life at the intersection of computer science and biology which, for many, is more like the intersection of science fiction and science, and explains how the unique properties of DNA could eventually enable us to store really big data in really small places for a really long time.

Microsoft blog editor

Artificial intelligence, Computer vision, Hardware and devices, Human-computer interaction, Programming languages and software engineering, Social sciences

2018 EMEA Scholarships to Advance Compelling Research in Healthcare, AI, Future of Work

The Microsoft Research PhD Scholarship Programme in EMEA awards scholarships to fund the most promising and groundbreaking computer science research by PhD students across Europe, the Middle East, and Africa. We are extremely proud today to highlight some of the selected projects for 2018.

Microsoft blog editor

Project Vermont

Graphics and multimedia, Hardware and devices, Human-computer interaction

Project Vermont: Improving camera performance through curved image sensors

By Richard Stoakley, Microsoft Research It’s hard to remember when our world wasn’t full of cameras. They’re in our phones and tablets, posted on street corners and in our buildings and cars. We rely on them not just to capture moments and memories, but for security, navigation, 3D spatial mapping and processing medical data. While […]

Microsoft blog editor