Microsoft Research Blog

Three mysteries in deep learning: Ensemble, knowledge distillation, and self-distillation 

January 19, 2021 | Zeyuan Allen-Zhu and Yuanzhi Li
Under now-standard techniques, such as over-parameterization, batch-normalization, and adding residual links, “modern age” neural network training—at least for image classification tasks and many others—is usually quite stable. Using standard neural network architectures and training algorithms (typically SGD with momentum), the learned models perform consistently well,…
diagram

Recent Posts

  1. diagram

    Three mysteries in deep learning: Ensemble, knowledge distillation, and self-distillation 

    January 19, 2021 | Zeyuan Allen-Zhu and Yuanzhi Li

    Under now-standard techniques, such as over-parameterization, batch-normalization, and adding residual links, “modern age” neural network training—at least for image classification tasks and many others—is usually quite stable. Using standard neural network architectures and training algorithms (typically SGD with momentum), the learned models perform consistently well,…

  2. VinVL: Advancing the state of the art for vision-language models 

    January 14, 2021 | Pengchuan Zhang, Lei Zhang, and Jianfeng Gao

    Humans understand the world by perceiving and fusing information from multiple channels, such as images viewed by the eyes, voices heard by the ears, and other forms of sensory input. One of the core aspirations in AI is to develop algorithms that endow computers with…

  4. graphical user interface

    Unadversarial examples: Designing objects for robust vision 

    December 22, 2020 | Hadi Salman

    Many of the items and objects we use in our daily lives were designed with people in mind. In October, the Reserve Bank of Australia put out into the world its redesigned $100 banknote. Some design elements remained the same—such as color and size, characteristics…

  6. ‘Seeing’ on tiny battery-powered microcontrollers with RNNPool 

    December 11, 2020

    Computer vision has rapidly evolved over the past decade, allowing for such applications as Seeing AI, a camera app that describes aloud a person’s surroundings, helping those who are blind or have low vision; systems that can detect whether a product, such as a computer…

  9. Image from Game of Drones simulation

    Research Collection – Reinforcement Learning at Microsoft 

    December 7, 2020

    Reinforcement learning is about agents taking information from the world and learning a policy for interacting with it, so that they perform better. So, you can imagine a future where, every time you type on the keyboard, the keyboard learns to understand you better. Or…

  11. A Microsoft custom data type for efficient inference 

    December 2, 2020

    AI is taking on an increasingly important role in many Microsoft products, such as Bing and Office 365. In some cases, it’s being used to power outward-facing features like semantic search in Microsoft Word or intelligent answers in Bing, and deep neural networks (DNNs) are…

Explore More

  • Events & conferences

    Events & conferences 

    Meet our community of researchers, learn about exciting research topics, and grow your network

  • Podcasts

    Podcasts 

    Ongoing conversations at the cutting edge of research

  • Microsoft Research Forum

    Microsoft Research Forum 

    Join us for a continuous exchange of ideas about research in the era of general AI