Search and find math solutions from a handwritten problem. Users snap a photo of the problem, and the app provides a step-by-step explanation, graphs, related examples, and video lectures to better understand the math problem.Explore the Microsoft Math app
When students are learning math, searching for relevant material can be challenging - typing an equation with complex symbols is cumbersome, and finding learning materials such as step-by-step techniques and videos is time-consuming. Students need self-learning tools that are convenient to use on mobile devices and connected to related resources from the web.
Leveraging advances in computer vision and language understanding, we wanted to build a seamless experience that would deliver a rich math learning experience to students, starting with a simple photo of the math problem on their mobile devices.
Part of Microsoft Math and the Bing application, the math service uses optical character recognition (OCR) to read a photo of a handwritten problem, solving the challenge of typing in complex equations. The math solver engine, hosted on Azure, generates step-by-step explanations and interactive graphs. To enhance educational value, powerful Bing Search APIs provide related content, online video lectures, and practice problems.
Solving problems from algebra to calculus
Microsoft Math uses OCR for handwriting to extract the problem from a student’s photo of their notes. The problem is standardized, classified, and solved. The answer is returned along with steps, lessons, and similar problems.
Technical details for Microsoft Math
The solution consists of a mobile experience powered by an intelligent, geo-distributed backend service hosted on Azure. The mobile experience is designed in React-Native to make the development cross-platform easier.
We built a math solver backend using the same math solver engine which is also integrated in OneNote. This is a compute-intensive service which is hosted on Azure cloud and geo-distributed across data centers to provide a performant math experience on the client.
In addition to the solver, we developed and trained a math language to understand the problem type and trigger appropriate answer modules. Using Bing Search APIs, which is available as part of Azure Cognitive Services, we can analyze the math problem, and provide relevant math concepts and video answers.
Sketch2Code converts hand-written drawings to HTML prototypes. Designers share ideas on a whiteboard, then changes are shown instantly in the browser—helping improve collaboration between the designer, developer, and customer.
Gen Studio is a prototype created with collaborators from The Metropolitan Museum of Art, Microsoft, and MIT. Gen Studio uses AI to visually navigate The Met’s art collection.
Make search more accurate and capable by using vector search. It leverages approximate nearest neighbor (ANN) algorithms to search through billions of deep learning model insights in milliseconds.
Snip Insights helps users find intelligent insights from a snip or screenshot. AI services convert a captured image to translated text, automatically detecting and tagging image content.
Explore the possibilities of AI
Jumpstart your own AI innovations with learning resources and development solutions from Microsoft AI.
Innovation Developer Hub
Explore insights and behind-the-scenes technology for breakthrough AI innovations. From Tech Minutes videos to Technology Deep Dives, learn about the engineering that powers the future of AI.
Learn to create your own AI experiences with learning paths in conversational AI, machine learning, AI for devices, cognitive services, autonomous systems, AI strategy, and more.
Start building AI solutions with powerful tools and services. Microsoft AI is a robust framework for developing AI solutions in conversational AI, machine learning, data sciences, robotics, IoT, and more.