AI Lab projects
Learn about breakthrough AI innovation with hands-on labs, code resources, and deep dives.
Microsoft Math
Search and find math solutions from a handwritten problem. Users snap a photo of the problem, and the app provides a step-by-step explanation, graphs, related examples, and video lectures to better understand the math problem.
Explore the Microsoft Math app
The need
When students are learning math, searching for relevant material can be challenging - typing an equation with complex symbols is cumbersome, and finding learning materials such as step-by-step techniques and videos is time-consuming. Students need self-learning tools that are convenient to use on mobile devices and connected to related resources from the web.
The idea
Leveraging advances in computer vision and language understanding, we wanted to build a seamless experience that would deliver a rich math learning experience to students, starting with a simple photo of the math problem on their mobile devices.
The solution
Part of Microsoft Math and the Bing application, the math service uses optical character recognition (OCR) to read a photo of a handwritten problem, solving the challenge of typing in complex equations. The math solver engine, hosted on Azure, generates step-by-step explanations and interactive graphs. To enhance educational value, powerful Bing Search APIs provide related content, online video lectures, and practice problems.
Solving problems from algebra to calculus
Microsoft Math uses OCR for handwriting to extract the problem from a student’s photo of their notes. The problem is standardized, classified, and solved. The answer is returned along with steps, lessons, and similar problems.
Technical details for Microsoft Math
The solution consists of a mobile experience powered by an intelligent, geo-distributed backend service hosted on Azure. The mobile experience is designed in React-Native to make the development cross-platform easier.
We built a math solver backend using the same math solver engine which is also integrated in OneNote. This is a compute-intensive service which is hosted on Azure cloud and geo-distributed across data centers to provide a performant math experience on the client.
In addition to the solver, we developed and trained a math language to understand the problem type and trigger appropriate answer modules. Using Bing Search APIs, which is available as part of Azure Cognitive Services, we can analyze the math problem, and provide relevant math concepts and video answers.
Resources:
Sketch2Code
Sketch2Code converts hand-written drawings to HTML prototypes. Designers share ideas on a whiteboard, then changes are shown instantly in the browser—helping improve collaboration between the designer, developer, and customer.
Gen Studio
Gen Studio is a prototype created with collaborators from The Metropolitan Museum of Art, Microsoft, and MIT. Gen Studio uses AI to visually navigate The Met’s art collection.
Vector search
Make search more accurate and capable by using vector search. It leverages approximate nearest neighbor (ANN) algorithms to search through billions of deep learning model insights in milliseconds.
Snip Insights
Snip Insights helps users find intelligent insights from a snip or screenshot. AI services convert a captured image to translated text, automatically detecting and tagging image content.
Explore the possibilities of AI
Make artificial intelligence real for your business today.
Create innovative AI solutions
Discover Azure AI—a portfolio of AI services designed for developers and data scientists. Take advantage of the decades of breakthrough research, responsible AI practices, and flexibility that Azure AI offers to build and deploy your own AI solutions.
