Application samples from AI Lab
Discover innovative applications and learn how to extend your projects with AI.
Sketch2Code uses AI to convert hand-written drawings to working HTML prototypes. Designers share ideas on a whiteboard, then changes are shown in the browser instantly.Try out Sketch2Code
Creative design process begins with collaboration on a whiteboard where designers share ideas. Once a design is drawn, it is usually manually translated into a working HTML wireframe. This takes time and delays the design process.
We can use Computer Vision to build a system that understands what a designer has drawn on a whiteboard, then translates that understanding to HTML code. This way we can generate HTML code directly from a hand-drawn image.
Custom Vision service trains models to detect HTML objects, then uses text recognition to extract handwritten text in the design. By combining the object and the text, we can generate HTML snippets of different design elements.
Dynamic design with Sketch2Code
Designers can collaborate and draw on a whiteboard—then see the resulting code to test their new web page ideas immediately.
Technical details for Sketch2Code
Computer Vision Service
Computer Vision is a discipline inside artificial intelligence that gives an application the capability to see and understand what it is seeing. Using Microsoft Cognitive Services, we can train Custom Computer Vision with millions of images and enable object detection for a wide range of types of objects.
In this case, we trained the model to recognize hand-drawn web design elements like a textbox or button. We use the text recognition functionality present in the Computer Vision Service to extract hand-written text present in the design. By combining the design element and the extracted content, we can generate HTML snippets of the different elements in the design. We then can infer the layout of the design from the position of the identified elements and generate the final HTML code.
Part of Azure Cognitive Services, the Custom Vision and Computer Vision APIs are used for object detection, OCR including handwriting recognition. With the Azure Cloud Platform, the objects returned help to identify the layout. Layouts are passed to the Sketch2Code MVC web application to generate HTML.
Explore the possibilities of AI
Find demos to get more ideas or learn about AI technology to jumpstart your own development.
Start creating your own AI experiences with courses in AI technology. Learn about conversational AI, machine learning, AI for devices, and cognitive services.
Dive into interactive demos that showcase AI in simple examples that explain the various capabilities of the Microsoft AI platform.