{"id":709066,"date":"2020-12-02T11:08:35","date_gmt":"2020-12-02T19:08:35","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=709066"},"modified":"2023-10-13T11:32:30","modified_gmt":"2023-10-13T18:32:30","slug":"a-microsoft-custom-data-type-for-efficient-inference","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/a-microsoft-custom-data-type-for-efficient-inference\/","title":{"rendered":"A Microsoft custom data type for efficient inference"},"content":{"rendered":"\n
\"animated<\/figure>\n\n\n\n

AI is taking on an increasingly important role in many Microsoft products, such as Bing and Office 365. In some cases, it\u2019s being used to power outward-facing features like semantic search in Microsoft Word or intelligent answers in Bing, and deep neural networks (DNNs) are one key to powering these features. One aspect of DNNs is inference\u2014once these networks are trained, they use inference to make judgments about unknown information based on prior learning.<\/p>\n\n\n\n

In Bing, for example, DNN inference enables multiple search scenarios including feature extraction, captioning, question answering, and ranking, which are all important tasks for customers to get accurate, fast responses to their search queries. These scenarios in Bing have stringent latency requirements and need to happen at an extremely large scale. As these inference scenarios and associated models grow in complexity and scale, the resources needed for inference are growing too\u2014between 3\u20135 times per year (see Figure 1).<\/p>\n\n\n\n

\"A
Figure 1: State-of-the-art models growth in size and cost, shown above as DNN inference necessary for models from 2011 until now. In five years\u2019 time since ResNet-50, models are showing an exponential growth in inference operations.<\/figcaption><\/figure>\n\n\n\n
\n\t
\n\t\t
\n\t\t\t\t\t\tPUBLICATION<\/span>\n\t\t\tPushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point<\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n

This growth trend line underscores the importance of minimizing the cost of inference for Microsoft. Data types\u2014that is, the formats used to represent data\u2014are a key factor in the cost of storage, access, and processing of the large quantities of data involved in deep learning models. In this blog post, we present a brief introduction to MSFP, a new class of data types optimized for efficient DNN inferencing, and how it is used in Project Brainwave<\/a> to provide low-cost inference at production scale. Project Brainwave and its technology play a critical role in powering the infrastructure for AI at Scale<\/a>, the Microsoft initiative to incorporate next-generation AI into its products and AI platforms. Our latest work is detailed in a paper accepted at the 34th Conference on Neural Information Processing Systems (NeurIPS 2020)<\/a>, titled \u201cPushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point<\/a>.\u201d<\/p>\n\n\n\n

\n\t
\n\t\t
\n\t\t\t\t\t\t\t\n\t\t\t\t\t\"illustrated\t\t\t\t<\/a>\n\t\t\t\t\t\t\tEvent<\/span>\n\t\t\tMicrosoft at NeurIPS 2020<\/span> <\/span><\/a>\t\t\t\t\t\t\t

Check out Microsoft's presence at NeurIPS 2020, including links to all of our NeurIPS publications, the Microsoft session schedule, and links to open career opportunities. <\/p>\n\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n

Microsoft Floating Point provides higher accuracy with low cost<\/h2>\n\n\n\n

Choosing how to represent numeric values in a computing platform is a trade-off between accuracy and cost. Using fewer bits per value, or moving from a floating-point to a fixed-point or integer format, decreases storage and computation costs but typically results in lower accuracy, that is, lower quality of results. MSFP is a new class of data types that deliver a fundamental shift in this trade-off, providing higher accuracy using fewer bits per value than traditional floating-point or integer formats. In particular, MSFP enables dot product operations\u2014the core of the matrix-matrix and matrix-vector multiplication operators critical to DNN inference\u2014to be performed nearly as efficiently as with integer data types, but with accuracy comparable to floating point.<\/p>\n\n\n\n

Floating-point formats capture a large dynamic range by using an exponent field to scale the significance of the value captured by the mantissa bits. Unfortunately, this exponent field requires additional bits to be stored with each value, and greatly complicates the process of adding or multiplying values. Eliminating the exponent, resulting in a fixed-point or integer representation, reduces computation and storage costs but at a potentially significant reduction in accuracy due to the loss of dynamic range. MSFP delivers the dynamic range of floating point at close to the cost of integer formats by associating a single exponent with a group, or \u201cbounding box\u201d, of mantissa values. MSFP is a class of data types parameterized by the number of mantissa and exponent bits and the bounding box size (see Figure 2). In this post, we\u2019ll focus on two versions of the data type, MSFP12 and MSFP16, which represent two points in the cost\/performance trade-off spectrum for MSFP types.<\/p>\n\n\n\n

\"MSFP
Figure 2: MSFP uses a shared exponent to achieve the dynamic range of floating-point formats such as fp32 and bfloat16 while keeping storage and computation costs close to those of integer formats.<\/figcaption><\/figure>\n\n\n\n

Through the coevolution of hardware design and algorithms, MSFP16 achieves 3x lower cost compared to Bfloat16, and MSFP12 achieves 4x lower cost compared to industry standard INT8 while delivering a comparable or better accuracy. Figure 3 shows the high-level overview of a systolic tensor core architecture containing multiple MSFP dot product units. Each dot product unit has a significantly lower circuit footprint compared to conventional float due to the shared exponent. We refer to the granularity at which the exponent is being shared between tensor elements as \u201cbounding box size.\u201d The math per bounding box is mostly performed in fixed-point format and the cost of dynamic scaling is amortized over the number of elements in each bounding box. To hide this complexity from software, the bounding box exponents are computed and updated in real time in hardware.<\/p>\n\n\n\n

\"Figure
Figure 3: Systolic tensor core architecture containing multiple MSFP dot product units. The multipliers and adders inside the dot product unit, shown on the right, operate on simple fixed-point values within a common bounding box. Only the accumulator on the far right deals with the more complex scaling issues associated with floating-point computation.<\/figcaption><\/figure>\n\n\n\n

Variants of MSFP together form a new Pareto frontier for computational performance\/mm2<\/sup> compared to a collection of competitive data types on commodity hardware. Figure 4 shows the accuracy\u2013area trade-off of using different data types for serving ResNet50 (ImageNet). MSFP outperforms existing data types in terms of area and energy cost while the model is held to a fixed accuracy.<\/p>\n\n\n\n

\"Figure
Figure 4: MSFP significantly improves upon previous data types in computational efficiency at each fixed level of accuracy. Left: relative area and energy cost of multiply-accumulate (MAC) using different data types on the same silicon. Right: ImageNet accuracy for ResNet-50 plotted versus normalized area cost. The area costs are normalized to Float32.<\/figcaption><\/figure>\n\n\n\n
\n\t
\n\t\t
\n\t\t\t\t\t\tBLOG<\/span>\n\t\t\tMicrosoft Turing Universal Language Representation model, T-ULRv2, tops XTREME leaderboard<\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n

MSFP is integrated within a mature large-scale production pipeline and has been used to ship various models empowering major online scenarios such as web search, question-answering, and image classification<\/a>. We further corroborated the efficacy of MSFP data type for inferencing various open-source benchmarks from different classes of deep learning models including CNNs, RNNs, and Transformers. Please refer to Table 1 below and the paper <\/a>for more details.<\/p>\n\n\n\n
Models<\/strong><\/th>Float32<\/strong>MSFP16<\/strong><\/th>MSFP15<\/strong><\/th>MSFP14<\/strong><\/th>MSFP13<\/strong><\/th>MSFP12<\/strong><\/th><\/tr><\/thead>
Resnet-50<\/td>1.000 (75.26)<\/td>1.000<\/td>0.999<\/td>0.994<\/strong><\/td>0.989<\/td>0.967<\/td><\/tr>
Resnet-101<\/td>1.000 (76.21)<\/td>1.000<\/td>1.000<\/td>0.998<\/td>0.991<\/strong><\/td>0.964<\/td><\/tr>
Resnet-152<\/td>1.000 (76.58) <\/td>1.000<\/td>1.001<\/td>0.997<\/td>0.991<\/strong><\/td>0.968<\/td><\/tr>
Inception-v3<\/td>1.000 (77.98)<\/td>1.000<\/td>1.005<\/td>1.001<\/td>0.990<\/strong><\/td>0.943<\/td><\/tr>
Inception-v4<\/td>1.000 (80.18)<\/td>1.000<\/td>1.001<\/td>1.000<\/td>0.993<\/strong><\/td>0.963<\/td><\/tr>
MobileNet-V1<\/td>1.000 (70.90) <\/td>0.998<\/td>0.997<\/td>0.990<\/strong><\/td>0.965<\/td>0.863<\/td><\/tr>
VGG16<\/td>1.000 (70.93)<\/td>1.000<\/td>1.004<\/td>1.005<\/td>1.003<\/td>1.002<\/strong><\/td><\/tr>
VGG19<\/td>1.000 (71.02)<\/td>1.000<\/td>1.002<\/td>1.001<\/td>1.002<\/td>1.000<\/strong><\/td><\/tr>
EfficientNet-S<\/td>1.000 (77.61)<\/td>1.000<\/td>0.998<\/td>0.992<\/strong><\/td>0.979<\/td>0.949<\/td><\/tr>
EfficientNet-M<\/td>1.000 (78.98)<\/td>1.000<\/td>0.998<\/td>0.993<\/strong><\/td>0.980<\/td>0.950<\/td><\/tr>
EfficientNet-L<\/td>1.000 (80.47)<\/td>1.000<\/td>0.999<\/td>0.993<\/strong><\/td>0.974<\/td>0.945<\/td><\/tr>
RNN-DR<\/td>1.000 (76.10) <\/td>1.000<\/td>1.008<\/td>1.003<\/td>1.009<\/td>1.000<\/strong><\/td><\/tr>
RNN-DS<\/td>1.000 (73.10)<\/td>1.000<\/td>1.012<\/td>1.005<\/td>1.022<\/td>0.992<\/strong><\/td><\/tr>
BERT-MRPC<\/td>1.000 (88.39)<\/td>1.000<\/td>1.005<\/td>1.002<\/td>1.008<\/td>1.018<\/strong><\/td><\/tr>
BERT-SQuAD1.1<\/td>1.000 (88.45)<\/td>1.000<\/td>0.998<\/td>0.998<\/td>0.997<\/td>0.990<\/strong><\/td><\/tr>
BERT-SQuADv2<\/td>1.000 (77.23)<\/td>1.000<\/td>0.999<\/td>0.999<\/td>0.993<\/strong><\/td>0.989<\/td><\/tr>
Memory density<\/td>1.0x<\/td>3.8x<\/td>4.3x<\/td>4.9x<\/td>5.8x<\/td>7.1x<\/td><\/tr>
Arithmetic density<\/td>1.0x<\/td>8.8x<\/td>10.8x<\/td>13.9x<\/td>18.3x<\/td>31.9x<\/td><\/tr><\/tbody><\/table>
Table 1: Normalized accuracy of different benchmarks across a range of bit-widths. The values are normalized with respect to the Float32 model accuracy listed in column 2 in parentheses. Configurations with the lowest bit-width that stay within 1% of Float32 accuracy are shown in bold.<\/figcaption><\/figure>\n\n\n\n

Project Brainwave takes MSFP to production<\/h2>\n\n\n\n

Microsoft leverages the benefits of MSFP via Project Brainwave, an internally developed system for real-time production-scale DNN inference in the cloud. Project Brainwave is used heavily in Bing and for our Office 365 workloads. Figure 5 shows a high-level view of the Project Brainwave Neural Processing Unit (NPU) architecture. The most critical part of the NPU is the matrix-vector multiplier (MVM) shown in the upper left. The MVM unit and associated matrix register file (MRF) and vector register file (VRF) are designed from the ground up to exploit the MSFP format.<\/p>\n\n\n\n

\"diagram\"<\/a>
Figure 5: Project Brainwave NPU Architecture. The thicker arrows represent the datapath connections, which carry data between the MSFP-based matrix-vector multiplier (MVM) and other function units. Data is routed via crossbar (XBAR) units. Firmware running on the control processor issues commands to the instruction dispatcher, which directs the crossbars and function units via control signals (light arrows). For more details see \u201cA Configurable Cloud-Scale DNN Processor for Real-Time AI.<\/a>\u201d<\/figcaption><\/figure>\n\n\n\n

Project Brainwave builds on the field-programmable gate arrays (FPGAs) deployed in Microsoft data centers as part of Project Catapult<\/a>. The fine-grain bit-level configurability of FPGAs allows us to harvest the increased computational, storage, and bandwidth efficiencies of each incremental reduction in the bit width of the values on which we operate, which is critical for taking advantage of MSFP. FPGA flexibility has also enabled us to experiment and push the boundaries of low-precision computation for DNN inference. We were able to deploy MSFP to production at low risk because the parameters of our MSFP implementation can be adjusted simply by re-synthesizing our FPGA configuration.<\/p>\n\n\n\n

Hardening MSFP in custom silicon<\/h2>\n\n\n\n

Now that MSFP has been proven in large-scale production systems, a natural next step is to consider hardening the performance-critical logic into custom silicon. Working together with our partners in the Intel Programmable Solutions Group (PSG), we\u2019ve delivered significant improvement in area and energy efficiency through silicon hardening\u2014resulting in the industry\u2019s first AI-accelerated FPGA using MSFP tensor cores. The Intel Stratix 10 NX device represents the first generation of its kind bringing together Microsoft algorithms embedded into hardened silicon while continuing to harness the advantages of flexible reconfigurable hardware.<\/p>\n\n\n\n

In the Stratix 10 NX device, over 4,000 custom digital signal processing (DSP) blocks deliver over 250 peak teraflops of MSFP. These hardened units are uniquely architected to blend seamlessly with reconfigurable logic, supporting flexible dataflows that can be finetuned for target applications\u2014for example, the ability to scale up and down tensor cores while supporting varying degrees of sparsity. Given the rapid change in state-of-the-art inference algorithms, the NX enables Microsoft to continue evolving its hardware platform to handle and accelerate best-of-breed AI algorithms (without the drawbacks of multi-year silicon turnarounds).<\/p>\n\n\n\n

Project Brainwave gains momentum for next-gen technologies<\/h2>\n\n\n\n

MSFP represents a major turning point in the journey that started in 2016 to deliver large-scale inferencing capabilities for Microsoft. Our understanding and research of low-precision arithmetic has advanced by leaps and bounds, with our confidence strengthened by the success of dozens of state-of-the-art AI models that have been deployed for critical scenarios in Bing and Office 365 using MSFP.<\/p>\n\n\n\n

Project Brainwave, a platform leveraging reconfigurable logic, has provided the critical and flexible hardware platform to evolve algorithms with confidence. As research in low precision and sparsity continues to advance, we expect Project Brainwave\u2014turbocharged by hardened MSFP\u2014will continue to play a pivotal role in enabling us to deliver the next wave of advantages through algorithm codesign in hardware. We\u2019re extremely excited for what\u2019s coming next.<\/p>\n\n\n\n

Acknowledgments<\/h3>\n\n\n\n

Microsoft Floating Point, and this work specifically, would not be possible without the broad team of engineers and researchers involved. We\u2019d particularly like to thank Daniel Lo<\/a> and Ritchie Zhao for their critical contributions, as well as Ming Liu<\/a>, Jeremy Fowers<\/a>, Kalin Ovtcharov<\/a>, Anya Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, Alessandro Forin, Haishan Zhu, Taesik Na, Prerak Patel, Shuai Che, Ahmad El Husseini, Lok Chand Koppaka, Xia Song, Subhojit Som, and Kaustav Das. We also thank Maximilian Golub, Rasoul Shafipour, and Greg Yang for their feedback on the draft of the paper. We thank Raja Venugopal for his support of this work.<\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

AI is taking on an increasingly important role in many Microsoft products, such as Bing and Office 365. In some cases, it\u2019s being used to power outward-facing features like semantic search in Microsoft Word or intelligent answers in Bing, and deep neural networks (DNNs) are one key to powering these features. One aspect of DNNs […]<\/p>\n","protected":false},"author":38838,"featured_media":709474,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[{"type":"user_nicename","value":"Bita Darvish Rouhani","user_id":"37839"},{"type":"user_nicename","value":"Doug Burger","user_id":"31582"},{"type":"user_nicename","value":"Eric Chung","user_id":"31746"},{"type":"user_nicename","value":"Rangan Majumder","user_id":"38931"},{"type":"user_nicename","value":"Sangeetha Shekar","user_id":"39907"},{"type":"user_nicename","value":"Saurabh Tiwary","user_id":"39603"},{"type":"user_nicename","value":"Sitaram Lanka","user_id":"37485"},{"type":"user_nicename","value":"Steve Reinhardt","user_id":"37488"}],"msr_hide_image_in_river":0,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556,13552,13545],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-709066","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-research-area-hardware-devices","msr-research-area-human-language-technologies","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[649749,486102,171431],"related-events":[708199],"related-researchers":[{"type":"user_nicename","value":"Doug Burger","user_id":31582,"display_name":"Doug Burger","author_link":"Doug Burger<\/a>","is_active":false,"last_first":"Burger, Doug","people_section":0,"alias":"dburger"},{"type":"user_nicename","value":"Rangan Majumder","user_id":38931,"display_name":"Rangan Majumder","author_link":"Rangan Majumder<\/a>","is_active":false,"last_first":"Majumder, Rangan","people_section":0,"alias":"ranganm"},{"type":"user_nicename","value":"Sangeetha Shekar","user_id":39907,"display_name":"Sangeetha Shekar","author_link":"Sangeetha Shekar<\/a>","is_active":false,"last_first":"Shekar, Sangeetha","people_section":0,"alias":"sshekar"},{"type":"user_nicename","value":"Sitaram Lanka","user_id":37485,"display_name":"Sitaram Lanka","author_link":"Sitaram Lanka<\/a>","is_active":false,"last_first":"Lanka, Sitaram","people_section":0,"alias":"slanka"}],"msr_type":"Post","featured_image_thumbnail":"\"\"","byline":"","formattedDate":"December 2, 2020","formattedExcerpt":"AI is taking on an increasingly important role in many Microsoft products, such as Bing and Office 365. In some cases, it\u2019s being used to power outward-facing features like semantic search in Microsoft Word or intelligent answers in Bing, and deep neural networks (DNNs) are…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/709066","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/38838"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=709066"}],"version-history":[{"count":32,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/709066\/revisions"}],"predecessor-version":[{"id":976299,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/709066\/revisions\/976299"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/709474"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=709066"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=709066"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=709066"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=709066"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=709066"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=709066"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=709066"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=709066"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=709066"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=709066"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=709066"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}