{"id":717850,"date":"2021-01-19T13:26:10","date_gmt":"2021-01-19T21:26:10","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=717850"},"modified":"2021-03-08T13:04:49","modified_gmt":"2021-03-08T21:04:49","slug":"three-mysteries-in-deep-learning-ensemble-knowledge-distillation-and-self-distillation","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/three-mysteries-in-deep-learning-ensemble-knowledge-distillation-and-self-distillation\/","title":{"rendered":"Three mysteries in deep learning: Ensemble, knowledge distillation, and self-distillation"},"content":{"rendered":"\n
\"\"\/<\/figure>\n\n\n\n

Under now-standard techniques, such as over-parameterization, batch-normalization, and adding residual links, \u201cmodern age\u201d neural network training\u2014at least for image classification tasks and many others\u2014is usually quite stable. Using standard neural network architectures and training algorithms (typically SGD with momentum), the learned models perform consistently well, not only in terms of training accuracy but even in test accuracy, regardless of which random initialization or random data order is used during the training. For instance, if one trains the same WideResNet-28-10 architecture on the CIFAR-100 dataset 10 times with different random seeds, the mean test accuracy is 81.51% while the standard deviation is only 0.16%.<\/p>\n\n\n\n

\n\t
\n\t\t
\n\t\t\t\t\t\tPublication<\/span>\n\t\t\tTowards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning<\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n

In a new paper, \u201cTowards Understanding Ensemble, Knowledge Distillation, and Self-Distillation in Deep Learning<\/a>,\u201d we focus on studying the discrepancy of neural networks during the training process that has arisen purely from randomizations. We ask the following questions: besides this small deviation in test accuracies, do the neural networks trained from different random initializations actually learn very different functions? If so, where does the discrepancy come from? How do we reduce such discrepancy and make the neural network more stable or even better? These questions turn out to be quite nontrivial, and they relate to the mysteries of three techniques widely used in deep learning.<\/p>\n\n\n\n

Three of the mysteries in deep learning<\/h2>\n\n\n\n

Mystery 1: Ensemble. <\/strong>The learned networks \\(F_1\\),\u2026\\(F_{10}\\) using different random seeds\u2014despite having very similar test performance\u2014are observed to associate with very different functions. Indeed, using a well-known technique called ensemble, which merely takes the unweighted average of the outputs of these independently trained networks, one can obtain a huge boost in test-time performance in many deep learning applications. (See Figure 1 below.) This implies the individual functions \\(F_1\\),\u2026\\(F_{10}\\) must be different. However, why does ensemble work with a sudden performance boost? Alternatively, if one directly trains (\\(F_1\\)+\u22ef+\\(F_{10}\\))\/10 altogether, why does the performance boost disappear?<\/p>\n\n\n\n

\"F(1)
Figure 1: Ensemble gives a performance boost to test accuracies in deep learning applications, but such accuracy gains cannot be matched by training the average of the models directly.<\/figcaption><\/figure>\n\n\n\n

Mystery 2:<\/strong> Knowledge distillation. <\/strong>While ensemble is great for improving test-time performance, it becomes 10 times slower during inference time (that is, test time): we need to compute the outputs of 10 neural networks instead of one. This is an issue when we deploy such models in a low-energy, mobile environment. To fix it, a seminal technique called knowledge distillation (opens in new tab)<\/span><\/a> was proposed. That is, knowledge distillation <\/em>simply trains another individual model to match the output <\/em>of the ensemble. Here, the output of the ensemble (also called the dark knowledge<\/em>) on a cat image may look like \u201c80% cat + 10% dog + 10% car,\u201d while the true training label is \u201c100% cat.\u201d (See Figure 2 below.)<\/p>\n\n\n\n

It turns out the so-trained individual model can, to great extent, match the test-time performance of a 10-times-bigger ensemble. However, this leads to more questions. Why does matching the outputs of the ensemble give us better test accuracy compared to matching the true labels? Moreover, can we perform ensemble learning over the models after knowledge distillation to further improve test accuracy?<\/p>\n\n\n\n

\"This
Figure 2: Knowledge distillation and self-distillation also give performance boosts in deep learning.<\/figcaption><\/figure>\n\n\n\n

Mystery 3: Self-distillation.<\/strong> Note that knowledge distillation at least intuitively makes sense: the teacher ensemble model has 84.8% test accuracy, so the student individual model can achieve 83.8%. The following phenomenon, called self-distillation<\/em> (or \u201cBe Your Own Teacher (opens in new tab)<\/span><\/a>\u201d), is completely astonishing\u2014by performing knowledge distillation against an individual model of the same architecture, test accuracy can also be improved. (See Figure 2 above.) Consider this: if training an individual model only gives 81.5% test accuracy, then how come \u201ctraining the same model again using itself as the teacher<\/em>\u201d suddenly boosts the test accuracy consistently to 83.5%?<\/p>\n\n\n\n

Ensemble of neural networks versus ensemble of feature mappings<\/h2>\n\n\n\n

Most existing theories on ensemble only apply to the case where individual models are fundamentally different (for example, decision trees supported on different subsets of the variables) or trained over different datasets (such as bootstrapping). They cannot justify the aforementioned phenomenon in the deep learning world, where individually trained neural networks are of the same <\/em>architecture and using the same <\/em>training data\u2014their only difference comes from the randomness during training.<\/p>\n\n\n\n

Perhaps the existing theorem closest to matching ensemble in deep learning is the ensemble of random feature mappings<\/em>. On one hand, combining multiple linear models of random (prescribed) features should improve test-time performance because it increases the number of features. On the other hand, in certain parameter regimes, neural network weights can stay very close to their initializations (known as the neural tangent kernel, or NTK, regime), and the resulting network is merely learning a linear function over prescribed feature mappings that are completely decided by the random initialization (see this work<\/a>). When combining these two, one may conjecture that ensemble in deep learning shares the principle of ensemble in random feature mappings. That leads us to the following question:<\/p>\n\n\n\n

Does ensemble\/knowledge distillation work in the same way in deep learning compared to that in random feature mappings (namely, the NTK feature mappings)?<\/strong><\/p>\n\n\n\n

Answer: not really, as evidenced by the experiment in Figure 3 below.<\/strong> This figure compares ensemble and knowledge distillation in deep learning versus that in a linear model over random feature mappings. Ensemble works in both <\/em>cases. However, the accuracies in Figure 3 clearly show that they work for completely different <\/em>reasons. Specifically:<\/p>\n\n\n\n