Search and find mathematical solutions from a handwritten problem. Users snap a photo of the problem, and the app provides a step-by-step explanation, graphs, related examples and video lectures to better understand the mathematical problem.Explore the Microsoft Math app
When students are learning mathematics, searching for relevant material can be challenging – typing an equation with complex symbols is cumbersome, and finding learning materials such as step-by-step techniques and videos is time-consuming. Students need self-learning tools that are convenient to use on mobile devices and connected to related resources from the web.
Leveraging advances in computer vision and language understanding, we wanted to build a seamless experience that would deliver a rich mathematical learning experience to students, starting with a simple photo of the mathematics problem on their mobile devices.
Part of Microsoft Math and the Bing application, the mathematics service uses optical character recognition (OCR) to read a photo of a handwritten problem, solving the challenge of typing in complex equations. The mathematics solver engine, hosted on Azure, generates step-by-step explanations and interactive graphs. To enhance educational value, powerful Bing Search APIs provide related content, online video lectures and practice problems.
Solving problems from algebra to calculus
Microsoft Math uses OCR for handwriting to extract the problem from a student’s photo of their notes. The problem is standardised, classified and solved. The answer is returned along with steps, lessons and similar problems.
Technical details for Microsoft Math
The solution consists of a mobile experience powered by an intelligent, geo-distributed backend service hosted on Azure. The mobile experience is designed in React-Native to make the development cross-platform easier.
We built a mathematics solver backend using the same mathematics solver engine which is also integrated in OneNote. This is a compute-intensive service which is hosted on Azure cloud and geo-distributed across data centres to provide a performant mathematics experience on the client.
In addition to the solver, we developed and trained a mathematical language to understand the problem type and trigger appropriate answer modules. Using Bing Search APIs, which is available as part of Azure Cognitive Services, we can analyse the mathematical problem, and provide relevant mathematical concepts and video answers.
Sketch2Code converts hand-written drawings to HTML prototypes. Designers share ideas on a whiteboard, then changes are shown instantly in the browser – helping to improve collaboration between the designer, developer and customer.
Gen Studio is a prototype created with collaborators from The Metropolitan Museum of Art, Microsoft and MIT. Gen Studio uses AI to visually navigate The Met’s art collection.
Make search more accurate and capable by using vector search. It leverages approximate nearest neighbour (ANN) algorithms to search through billions of deep learning model insights in milliseconds.
Snip Insights helps users find intelligent insights from a snip or screenshot. AI services convert a captured image into translated text, automatically detecting and tagging image content.
Explore the possibilities of AI
Jump-start your own AI innovations with learning resources and development solutions from Microsoft AI.
Innovation Developer Hub
Explore insights and behind-the-scenes technology for breakthrough AI innovations. From Tech Minutes videos to Technology Deep Dives, learn about the engineering that powers the future of AI.
Learn to create your own AI experiences with learning paths in conversational AI, machine learning, AI for devices, cognitive services, autonomous systems, AI strategy, and more.
Start building AI solutions with powerful tools and services. Microsoft AI is a robust framework for developing AI solutions in conversational AI, machine learning, data sciences, robotics, IoT, and more.