{"id":683157,"date":"2020-08-11T10:00:18","date_gmt":"2020-08-11T17:00:18","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=683157"},"modified":"2020-12-09T14:08:39","modified_gmt":"2020-12-09T22:08:39","slug":"adversarial-robustness-as-a-prior-for-better-transfer-learning","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/adversarial-robustness-as-a-prior-for-better-transfer-learning\/","title":{"rendered":"Adversarial robustness as a prior for better transfer learning"},"content":{"rendered":"\n
\"\"\/<\/figure>\n\n\n\n

Editor\u2019s note: This post and its research are the collaborative efforts of our team, which includes <\/em>Andrew Ilyas<\/em> (opens in new tab)<\/span><\/a> (PhD Student, MIT), <\/em>Logan Engstrom<\/em> (opens in new tab)<\/span><\/a> (PhD Student, MIT), <\/em>Aleksander M\u0105dry<\/em> (opens in new tab)<\/span><\/a> (Professor at MIT), <\/em>Ashish Kapoor<\/em><\/a> (Partner Research Manager).<\/em><\/p>\n\n\n\n

In practical machine learning, it is desirable to be able to transfer learned knowledge from some \u201csource\u201d task to downstream \u201ctarget\u201d tasks. This is known as transfer learning\u2014a simple and efficient way to obtain performant machine learning models, especially when there is little training data or compute available for solving the target task. Transfer learning is very useful in practice. For example, transfer learning allows perception models on a robot<\/a> or other autonomous system<\/a> to be trained on a synthetic dataset generated via a high-fidelity simulator, such as AirSim (opens in new tab)<\/span><\/a>, and then refined on a small dataset collected in the real world.<\/p>\n\n\n\n

Transfer learning is also common in many computer vision tasks, including image classification and object detection, in which a model uses some pretrained representation as an \u201cinitialization\u201d to learn a more useful representation for the specific task in hand. In a recent collaboration with MIT, we explore adversarial robustness as a prior for improving transfer learning in computer vision. We find that adversarially robust models outperform their standard counterparts on a variety of downstream computer vision tasks.<\/p>\n\n\n\n

Read Paper<\/a>                     Code & Models (opens in new tab)<\/span><\/a> <\/strong><\/p>\n\n\n\n

\"Workflow
Figure 1: A depiction of transfer learning.<\/figcaption><\/figure><\/div>\n\n\n\n

In our work we focus on computer vision and consider a standard transfer learning pipeline: “ImageNet pretraining.” This pipeline trains a deep neural network on ImageNet (opens in new tab)<\/span><\/a>, then tweaks this pretrained model for another target task, ranging from image classification of smaller datasets to more complex tasks like object detection and image segmentation.<\/p>\n\n\n\n

Refining the ImageNet pretrained model can be done in several ways. In our work we focus on two common methods:<\/p>\n\n\n\n