{"id":728458,"date":"2021-02-25T10:07:20","date_gmt":"2021-02-25T18:07:20","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=728458"},"modified":"2021-02-25T10:10:23","modified_gmt":"2021-02-25T18:10:23","slug":"hexa-self-supervised-pretraining-with-hard-examples-improves-visual-representations","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/hexa-self-supervised-pretraining-with-hard-examples-improves-visual-representations\/","title":{"rendered":"HEXA: Self-supervised pretraining with hard examples improves visual representations"},"content":{"rendered":"\n
\"\"\/<\/figure>\n\n\n\n

Humans perceive the world through observing a large number of visual scenes around us and then effectively generalizing\u2014in other words, interpreting and identifying scenes they haven\u2019t encountered before\u2014without heavily relying on labeled annotations for every single scene. One of the core aspirations in artificial intelligence is to develop algorithms and techniques that endow computers with a strong generalization ability to learn only from raw pixel data to make sense of the visual world, which aligns more closely with how humans process visual information.<\/p>\n\n\n\n

\n\t
\n\t\t
\n\t\t\t\t\t\tBlOG<\/span>\n\t\t\tDeep InfoMax: Learning good representations through mutual information maximization<\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n

Currently, self-supervised pretraining <\/a>(SSP) is rising as an emerging research field, showing great success in approaching this problem. The goal of SSP is to learn general-purpose intermediate representations, with the expectation that the representations carry rich semantic or structural meanings and can be beneficial to a variety of practical downstream tasks.<\/p>\n\n\n\n

SSP methods rely heavily on data augmentation (opens in new tab)<\/span><\/a> to create different views of an image using image transformations (opens in new tab)<\/span><\/a>, such as random crop (with flip and resize), color distortion, and Gaussian blur. However, image transformations are agnostic to the pretraining objectives, and it remains unknown how to augment views specifically based on the pretraining tasks themselves or how they affect the generalization of the learned models.<\/p>\n\n\n\n

\n\t
\n\t\t
\n\t\t\t\t\t\tPublication<\/span>\n\t\t\tSelf-supervised Pre-training with Hard Examples Improves Visual Representations.<\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n

In this blog post, we first review SSP as a problem of predicting pseudo-labels\u2014labels derived from the given data itself rather than with the help of human annotations. Based on this problem, we describe how to generate hard examples (HEXA), a family of augmented views whose pseudo-labels are difficult to predict. This leads to a novel SSP framework, and we develop two novel algorithms that create state-of-the-art performance on well-established vision tasks based on this framework. Please check out our paper, \u201cSelf-supervised Pre-training with Hard Examples Improves Visual Representations<\/a>,” which elaborates on the details presented in this blog post.<\/p>\n\n\n\n

Self-supervision: A pseudo-label view<\/h2>\n\n\n\n

The self-supervised task (opens in new tab)<\/span><\/a> (also known as pretext task) leverages and exploits a variety of different weak signals existing intrinsically in images as pseudo-labels, maximizing the agreement between pseudo-labels and the learned representations. (These weak signals often come with the data for free.) For example, we might rotate images at random and train a model to predict how each input image is rotated.<\/p>\n\n\n\n

The rotation prediction task is made-up, so the actual accuracy is unimportant, like the treatment of auxiliary tasks where we are concerned mostly with downstream task recognition accuracy. We expect the model to learn high-quality representations for real-world tasks, such as constructing an object recognition classifier with very few labeled samples. Most state-of-the-art SSP methods focus on designing novel pretext objectives, broadly categorized into two types:<\/p>\n\n\n\n