Microsoft Journal

Microsoft Journal began with a simple but large question, what would a personal journaling experience be like if it was designed to be used with a digital pen and touch? No mouse or keyboard.

After a quick prototype showed value, the team asked the second big question, “What experiences become possible if software can recognize when I write a star beside a line of ink?” Or, more basically, what happens if an application has a semantic understanding of the way that people naturally write notes. Once the team built a very simple proof of concept that proved machine learning could delightfully detect a star and associate it with nearby writing, the incubation began in earnest.

These basic questions have driven the design of Journal. You can see their influence in the page-based canvas that feels so familiar, in the AI used to enable intuitive editing gestures like scratch-out to erase and instant lasso to select, and in the way Journal recognizes the methods people use to organize their content like stars, headings, lists, diagrams, etc. This all combines in an experience that lets people fluidly create and seamlessly edit.

The Applied Sciences Group (ASG) is an applied R&D team dedicated to creating the next generation of computer interaction technologies and experiences. The team is purpose built to explore a problem space like inking on Windows tablets and produce incubations like Microsoft Journal.

“I am so proud of this team. They approached notetaking from first principles and incubated an innovative and delightful experience that advances the state of the art of digital ink. Microsoft Journal uses AI to make sense of ink and understand the natural structure of your handwritten notes. It is a pioneer of the next generation of products that leverage computer aided reasoning to help people create and achieve in ways that have never been possible.” – Stevie Bathiche, Technical Fellow and leader of Applied Sciences, Microsoft

During their journey building and refining Journal, the team overcame challenges together across the breadth of artificial intelligence, UX design, software engineering, and hardware.

“Collecting unbiased data from around the world to train AI is normally difficult. Doing it just as the world locked down over the COVID-19 pandemic was a huge challenge,” said Jacques Chamberland, Principal Program Manager. “The only solution was to be patient and persevere.”

Adapting their machine learning models, built from snippets of annotated training data, to the real world required many months of refinement. Managing the performance of Journal so that the entire application can run locally on most Windows tablets was also a significant challenge.

ASG’s incubations are usually shared internally within Microsoft in collaboration with other teams to help Microsoft better understand the opportunities, challenges, and technical solutions in a given space.

Journal is different because it was designed for a specific audience: people who want to use their Windows devices primarily with touch and pen.

Microsoft Journal search experience interface

“We decided to share Journal outside Microsoft, so we could learn how it gets used, understand what works and what doesn’t, and incorporate lessons learned back into Journal and share with our partners across Microsoft. The Garage is a great path for teams like ours to get feedback on experiments and incubations like Journal,” said Nathan Luquetta-Fish, Director of New Interface UX. “It’s a great pleasure to see customers using and enjoying Journal.”

The Journal team learned how important it is to intentionally design for digital ink experiences, and that inking is more than just writing content – it’s also about annotating content across various formats like PDFs.

They learned that selecting content easily and intuitively, then editing or manipulating the content, was a hard problem to solve since, unlike text, ink is only loosely structured and people’s editing habits are very personal. But editing was extremely important to get right to improve the flow of the inking experience.

So how did Journal make it to product? The Microsoft Experiences + Devices leadership team reviewed the experience of using Windows with a pen and recognized that Journal could fill a gap. Even though Journal was an incubation (meaning it did not have the full breadth of features of a full product), it still earned a 4.3 rating in the Microsoft store. In addition, lots of positive feedback from Microsoft employees who tried it out was also helpful in showing Journal’s impact.

Microsoft Journal has secured further funding and a dedicated, full-time team to develop and support moving forward. It will be an app that comes in-the-box on Windows pen enabled devices in SV2.

It will also appear in the Pen Menu meaning that people will be able to jump right into Journal every time they undock their pen (on devices that support the pen undock event).