AI Lab projects
Learn about breakthrough AI innovation with hands-on labs, code resources, and deep dives.
Sketch2Code
Sketch2Code uses AI to convert hand-written drawings to working HTML prototypes. Designers share ideas on a whiteboard, then changes are shown in the browser instantly.
Try out Sketch2Code
The need
Creative design process begins with collaboration on a whiteboard where designers share ideas. Once a design is drawn, it is usually manually translated into a working HTML wireframe. This takes time and delays the design process.
The idea
We can use Computer Vision to build a system that understands what a designer has drawn on a whiteboard, then translates that understanding to HTML code. This way we can generate HTML code directly from a hand-drawn image.
The solution
Custom Vision service trains models to detect HTML objects, then uses text recognition to extract handwritten text in the design. By combining the object and the text, we can generate HTML snippets of different design elements.
Dynamic design with Sketch2Code
Designers can collaborate and draw on a whiteboard—then see the resulting code to test their new web page ideas immediately.
Technical details for Sketch2Code
Computer Vision Service
Computer Vision is a discipline inside artificial intelligence that gives an application the capability to see and understand what it is seeing. Using Microsoft Cognitive Services, we can train Custom Computer Vision with millions of images and enable object detection for a wide range of types of objects.
In this case, we trained the model to recognize hand-drawn web design elements like a textbox or button. We use the text recognition functionality present in the Computer Vision Service to extract hand-written text present in the design. By combining the design element and the extracted content, we can generate HTML snippets of the different elements in the design. We then can infer the layout of the design from the position of the identified elements and generate the final HTML code.
Sketch2Code architecture
Part of Azure Cognitive Services, the Custom Vision and Computer Vision APIs are used for object detection, OCR including handwriting recognition. With the Azure Cloud Platform, the objects returned help to identify the layout. Layouts are passed to the Sketch2Code MVC web application to generate HTML.
Snip Insights
Snip Insights helps users find intelligent insights from a snip or screenshot. AI services convert a captured image to translated text, automatically detecting and tagging image content.
Microsoft Math
Microsoft Math uses optical character recognition (OCR) for handwriting to extract a math equation from a student’s photo of their notes. The problem is then standardized, classified, solved, and returned with solution steps and similar problems.
Explore the possibilities of AI
Make artificial intelligence real for your business today.
Create innovative AI solutions
Discover Azure AI—a portfolio of AI services designed for developers and data scientists. Take advantage of the decades of breakthrough research, responsible AI practices, and flexibility that Azure AI offers to build and deploy your own AI solutions.
