{"id":421017,"date":"2017-08-22T12:01:42","date_gmt":"2017-08-22T19:01:42","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=421017"},"modified":"2018-08-16T16:48:33","modified_gmt":"2018-08-16T23:48:33","slug":"microsoft-unveils-project-brainwave","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/microsoft-unveils-project-brainwave\/","title":{"rendered":"Microsoft unveils Project Brainwave for real-time AI"},"content":{"rendered":"

\"HotBy Doug Burger<\/a>, Distinguished Engineer, Microsoft<\/em><\/p>\n

Today at Hot Chips 2017 (opens in new tab)<\/span><\/a>, our cross-Microsoft team unveiled a new deep learning acceleration platform, codenamed Project Brainwave.\u00a0 I\u2019m delighted to share more details in this post, since Project Brainwave achieves a major leap forward in both performance and flexibility for cloud-based serving of deep learning models. We designed the system for real-time AI, which means the system processes requests as fast as it receives them, with ultra-low latency.\u00a0 Real-time AI is becoming increasingly important as cloud infrastructures process live data streams, whether they be search queries, videos, sensor streams, or interactions with users.<\/p>\n

The Project Brainwave system is built with three main layers:<\/p>\n

    \n
  1. A high-performance, distributed system architecture;<\/li>\n
  2. A hardware DNN engine synthesized onto FPGAs; and<\/li>\n
  3. A compiler and runtime for low-friction deployment of trained models.<\/li>\n<\/ol>\n

    First, Project Brainwave leverages the massive FPGA infrastructure (opens in new tab)<\/span><\/a>that Microsoft has been deploying over the past few years.\u00a0 By attaching high-performance FPGAs directly to our datacenter network, we can serve DNNs as hardware microservices (opens in new tab)<\/span><\/a>, where a DNN can be mapped to a pool of remote FPGAs and called by a server with no software in the loop.\u00a0 This system architecture both reduces latency, since the CPU does not need to process incoming requests, and allows very high throughput, with the FPGA processing requests as fast as the network can stream them.<\/p>\n

    Second, Project Brainwave uses a powerful \u201csoft\u201d DNN processing unit (or DPU), synthesized onto commercially available FPGAs.\u00a0 A number of companies\u2014both large companies and a slew of startups\u2014are building hardened DPUs.\u00a0 Although some of these chips have high peak performance, they must choose their operators and data types at design time, which limits their flexibility.\u00a0 Project Brainwave takes a different approach, providing a design that scales across a range of data types, with the desired data type being a synthesis-time decision.\u00a0 The design combines both the ASIC digital signal processing blocks on the FPGAs and the synthesizable logic to provide a greater and more optimized number of functional units.\u00a0 This approach exploits the FPGA\u2019s flexibility in two ways.\u00a0 First, we have defined highly customized, narrow-precision data types that increase performance without real losses in model accuracy.\u00a0 Second, we can incorporate research innovations into the hardware platform quickly (typically a few weeks), which is essential in this fast-moving space. \u00a0As a result, we achieve performance comparable to – or greater than – many of these hard-coded DPU chips but are delivering the promised performance today.<\/p>\n

    \"Intel

    At Hot Chips, Project Brainwave was demonstrated using Intel\u2019s new 14 nm Stratix 10 FPGA.<\/p><\/div>\n

    Third, Project Brainwave incorporates a software stack designed to support the wide range of popular deep learning frameworks.\u00a0 We already support Microsoft\u00a0Cognitive Toolkit<\/a>\u00a0and Google\u2019s Tensorflow (opens in new tab)<\/span><\/a>, and plan to support many others.\u00a0 We have defined a graph-based intermediate representation, to which we convert models trained in the popular frameworks, and then compile down to our high-performance infrastructure.<\/p>\n

    We architected this system to show high actual<\/em> performance across a wide range of complex models, with batch-free execution.\u00a0 Companies and researchers building DNN accelerators often show performance demos using convolutional neural networks (CNNs).\u00a0 Since CNNs are so compute intensive, it is comparatively simple to achieve high performance numbers.\u00a0 Those results are often not representative of performance on more complex models from other domains, such as LSTMs or GRUs for natural language processing.\u00a0 Another technique that DNN processors often use to boost performance is running deep neural networks with high degrees of batching.\u00a0 While this technique is effective for throughput-based architectures\u2014as well as off-line scenarios such as training\u2014it is less effective for real-time AI.\u00a0 With large batches, the first query in a batch must wait for all of the many queries in the batch to complete.\u00a0 Our system, designed for real-time AI, can handle complex, memory-intensive models such as LSTMs (opens in new tab)<\/span><\/a>, without using batching to juice throughput.<\/p>\n

    At Hot Chips, Eric Chung<\/a> and Jeremy Fowers<\/a> demonstrated the Project Brainwave system ported to Intel\u2019s new 14 nm Stratix 10 FPGA (opens in new tab)<\/span><\/a>. You can view the PowerPoint deck they presented at the event here<\/a> (PDF file).<\/p>\n

    Even on early Stratix 10 silicon, the ported Project Brainwave system ran a large GRU model\u2014five times larger than Resnet-50\u2014with no batching, and achieved record-setting performance.\u00a0 The demo used Microsoft\u2019s custom 8-bit floating point format (\u201cms-fp8\u201d), which does not suffer accuracy losses (on average) across a range of models.\u00a0 We showed Stratix 10 sustaining 39.5 Teraflops<\/strong> on this large GRU, running each request in under one millisecond.\u00a0 At that level of performance, the Brainwave architecture sustains execution of over 130,000 compute operations per cycle, driven by one macro-instruction being issued each 10 cycles.\u00a0 Running on Stratix 10, Project Brainwave thus achieves unprecedented levels of demonstrated real-time AI performance on extremely challenging models.\u00a0 As we tune the system over the next few quarters, we expect significant further performance improvements.<\/p>\n

    We are working to bring this powerful, real-time AI system to users in Azure (opens in new tab)<\/span><\/a>, so that our customers can benefit from Project Brainwave directly, complementing the indirect access through our services such as Bing (opens in new tab)<\/span><\/a>.\u00a0 In the near future, we\u2019ll detail when our Azure customers will be able to run their most complex deep learning models at record-setting performance.\u00a0 With the Project Brainwave system incorporated at scale and available to our customers, Microsoft Azure will have industry-leading capabilities for real-time AI.<\/p>\n

     <\/p>\n

    Related:<\/strong><\/p>\n