Sketch2Code at AI Lab
Sketch2Code uses AI to convert hand-written drawings to working HTML prototypes. Designers share ideas on a whiteboard, then changes are shown in the browser instantly.Try out Sketch2Code
Creative design process begins with collaboration on a whiteboard where designers share ideas. Once a design is drawn, it is usually manually translated into a working HTML wireframe. This takes time and delays the design process.
We can use Computer Vision to build a system that understands what a designer has drawn on a whiteboard, then translates that understanding to HTML code. This way we can generate HTML code directly from a hand-drawn image.
Custom Vision service trains models to detect HTML objects, then uses text recognition to extract handwritten text in the design. By combining the object and the text, we can generate HTML snippets of different design elements.
Dynamic design with Sketch2Code
Designers can collaborate and draw on a whiteboard—then see the resulting code to test their new web page ideas immediately.
Technical details for Sketch2Code
Computer Vision Service
Computer Vision is a discipline inside artificial intelligence that gives an application the capability to see and understand what it is seeing. Using Microsoft Cognitive Services, we can train Custom Computer Vision with millions of images and enable object detection for a wide range of types of objects.
In this case, we trained the model to recognize hand-drawn web design elements like a textbox or button. We use the text recognition functionality present in the Computer Vision Service to extract hand-written text present in the design. By combining the design element and the extracted content, we can generate HTML snippets of the different elements in the design. We then can infer the layout of the design from the position of the identified elements and generate the final HTML code.
Part of Azure Cognitive Services, the Custom Vision and Computer Vision APIs are used for object detection, OCR including handwriting recognition. With the Azure Cloud Platform, the objects returned help to identify the layout. Layouts are passed to the Sketch2Code MVC web application to generate HTML.
Pix2Story uses Natural Language Processing (NLP) for storytelling. AI scans a picture, applies a writing style, and generates a story—demonstrating how AI can drive creativity.
Snip Insights helps users find intelligent insights from a snip or screenshot. AI services convert a captured image to translated text, automatically detecting and tagging image content.
Microsoft Math uses optical character recognition (OCR) for handwriting to extract a math equation from a student’s photo of their notes. The problem is then standardized, classified, solved, and returned with solution steps and similar problems.
Explore the possibilities of AI
Jumpstart your own AI innovations with learning resources and development solutions from Microsoft AI.
Learn to create your own AI experiences with courses in AI technology. Engage with learning paths in conversational AI, machine learning, AI for devices, cognitive services, autonomous systems, AI business strategies, and responsible AI.
Start building AI solutions with powerful tools and services. Microsoft AI is a robust framework for developing AI solutions in conversational AI, machine learning, data sciences, robotics, IoT, and more.