{"id":789503,"date":"2021-10-28T09:46:45","date_gmt":"2021-10-28T16:46:45","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=789503"},"modified":"2021-10-28T09:46:47","modified_gmt":"2021-10-28T16:46:47","slug":"acav100m-scaling-up-self-supervised-audio-visual-learning-with-automatically-curated-internet-videos","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/acav100m-scaling-up-self-supervised-audio-visual-learning-with-automatically-curated-internet-videos\/","title":{"rendered":"ACAV100M: Scaling up self-supervised audio-visual learning with automatically curated internet videos"},"content":{"rendered":"\n

The natural association between visual observations and their corresponding sounds has exhibited powerful self-supervision signals for learning video representations<\/a>, which makes the ever-growing amount of online video an attractive data source for self-supervised learning. However, online videos often provide imperfectly aligned audio-visual signals because of overdubbed audio; models trained on uncurated videos have been shown to learn suboptimal representations (opens in new tab)<\/span><\/a> due to the misalignment issues. Therefore, existing approaches rely almost exclusively on manually curated datasets (opens in new tab)<\/span><\/a> with a predetermined taxonomy of semantic concepts (opens in new tab)<\/span><\/a>, where there is a high chance of audio-visual correspondence. This severely limits the utility of online videos for self-supervised learning, which begs the question: How can we fully leverage online videos without extensive human effort?<\/p>\n\n\n\n

\"ACAV100M<\/figure>\n\n\n\n

Today, in collaboration with Seoul National University and NVIDIA Research, we are thrilled to release an automatic dataset curation pipeline and the largest video dataset for self-supervised audio-visual learning, dubbed\u00a0ACAV100M (opens in new tab)<\/span><\/a>\u00a0(\/\u02c8\u00e4r\u02cck\u012bv\/;\u00a0a<\/strong>utomatically\u00a0c<\/strong>urated\u00a0a<\/strong>udio-v<\/strong>isual dataset).\u00a0Our dataset is constructed from\u00a0uncurated\u00a0online videos at an unprecedented scale: We processed 140 million full-length videos (total duration 1,030 years,\u00a0consuming 2 petabytes of storage) and reduced it to 100 million clips (10 seconds each, 31 years in total) that exhibit the highest audio-visual correspondence. This is the largest video dataset in the\u00a0current\u00a0literature\u00a0and\u00a0is orders of magnitude larger than\u00a0existing datasets for audio-visual learning.\u00a0We\u00a0are\u00a0also releasing\u00a0the code (opens in new tab)<\/span><\/a>,\u00a0so\u00a0that\u00a0the community can build upon our pipeline to create\u00a0other\u00a0datasets for self-supervised learning.\u00a0We provide full details of\u00a0the automatic dataset curation\u00a0pipeline in our\u00a0ICCV 2021 (opens in new tab)<\/span><\/a>\u00a0paper \u201cACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual\u00a0Video\u00a0Representation Learning.<\/a>\u201d\u00a0<\/p>\n\n\n\n

Posing dataset creation as subset selection with maximum mutual information<\/h2>\n\n\n\n

We formulate a constrained optimization problem where the objective is to find a subset that maximizes the total mutual information between audio and visual channels in the videos. Mutual information (MI) tells us how much the knowledge of one variable reduces uncertainty about the other. A subset with the maximum MI is therefore likely to have a high portion of videos with audio-visual correspondence, which is necessary for self-supervised learning. <\/p>\n\n\n\n

One could estimate audio-visual MI per video, independently of each other, and create a subset that provides the maximum MI. This can be done, for example, via contrastive learning (opens in new tab)<\/span><\/a> using co-occurring audio-visual signals as positive pairs. Unfortunately, we encounter the \u201cchicken-and-egg\u201d problem: contrastive learning requires a dataset from which we can reliably construct the positive pairs, but that is exactly what we set out to find in the first place! The literature already suggests that models trained on uncurated videos suffer from suboptimal representations due to misaligned audio-visual signals (opens in new tab)<\/span><\/a>.  <\/p>\n\n\n\n

In this work, rather than estimating MI at the instance level, we turn to set-based MI estimation by Banerjee et al. (2005) (opens in new tab)<\/span><\/a> that quantifies the information shared by two clustering assignments of a dataset. Given a candidate subset, we cluster the videos with respect to audio and visual signals, respectively, and measure the MI from a contingency table that encodes the overlap between audio and visual clusters. We found this set-based approach to be more robust to real-world noise, which helps it create datasets with a higher audio-visual correspondence than instance-based MI estimation. <\/p>\n\n\n\n

Finding a global solution to our constrained optimization problem is NP-hard;\u00a0we implement a\u00a0greedy solution\u00a0widely used\u00a0in the\u00a0submodular\u00a0optimization literature (opens in new tab)<\/span><\/a>. Specifically, we use the batch greedy algorithm of\u00a0Chen and Krause (2013) (opens in new tab)<\/span><\/a>\u00a0that randomly samples a batch from the remaining pool of candidates and searches for the next element to be added to the solution only within that batch. This batch trick reduces the complexity from quadratic to linear in terms of the population size.\u00a0We implemented\u00a0several other techniques to make our whole pipeline scalable to hundreds of millions of video clips. We provide the full details of the pipeline\u00a0in our paper<\/a>.\u00a0\u00a0<\/p>\n\n\n\n

Evaluating the utility of datasets for self-supervised audio-visual learning<\/h2>\n\n\n\n
\"Graphs:
Figure 1. Linear evaluation on downstream tasks. The top-1\/5 accuracy (%) of video classification on UCF101, audio classification on ESC-50 and audio-visual classification on Kinetics-Sounds (KS). We group the results by the downstream tasks and by the scale of the pretrain datasets. Baseline datasets are Kinetics-Sounds (20K), VGG-Sound (200K), and AudioSet (2M). All models are identical and trained using the same protocol except for the datasets used to pretrain them.<\/figcaption><\/figure>\n\n\n\n

To evaluate the utility of our dataset in self-supervised audio-visual learning, we compare our dataset with popular video datasets widely used in the self-supervised learning literature. We pretrain identical models on different datasets via contrastive learning (opens in new tab)<\/span><\/a> and perform linear evaluation on standard benchmarks. The figure above shows that models pretrained on our datasets (\u201cclustering\u201d, green bars) perform as well as, and sometimes even outperform, the models pretrained on existing datasets (\u201cbaselines\u201d, pink bars) that involve human annotation or manual verification. This shows that datasets created by our approach indeed provide audio-visual correspondence necessary for self-supervised approaches. We provide in-depth analyses of the results in our paper (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

Looking forward<\/h2>\n\n\n\n

Large-scale datasets are the cornerstone of self-supervised representation learning. We hope that the video understanding community will find value in our ACAV100M (opens in new tab)<\/span><\/a> dataset, which we release in the form of public URLs so anyone can access the data as long as the videos are still available online. We are open sourcing the dataset creation pipeline (opens in new tab)<\/span><\/a>, which can be applied to other scenarios involving different modalities such as vision and language. We look forward to seeing rapid development in learning from unlabeled videos with our new dataset and tools.<\/p>\n","protected":false},"excerpt":{"rendered":"

The natural association between visual observations and their corresponding sounds has exhibited powerful self-supervision signals for learning video representations, which makes the ever-growing amount of online video an attractive data source for self-supervised learning. However, online videos often provide imperfectly aligned audio-visual signals because of overdubbed audio; models trained on uncurated videos have been shown […]<\/p>\n","protected":false},"author":40735,"featured_media":789698,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[{"type":"user_nicename","value":"Yale Song","user_id":"37422"}],"msr_hide_image_in_river":0,"footnotes":""},"categories":[1],"tags":[],"research-area":[13562],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-789503","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-computer-vision","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[694407],"related-projects":[],"related-events":[778099],"related-researchers":[],"msr_type":"Post","featured_image_thumbnail":"\"ACAV100M","byline":"Yale Song","formattedDate":"October 28, 2021","formattedExcerpt":"The natural association between visual observations and their corresponding sounds has exhibited powerful self-supervision signals for learning video representations, which makes the ever-growing amount of online video an attractive data source for self-supervised learning. However, online videos often provide imperfectly aligned audio-visual signals because of…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/789503","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/40735"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=789503"}],"version-history":[{"count":4,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/789503\/revisions"}],"predecessor-version":[{"id":789710,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/789503\/revisions\/789710"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/789698"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=789503"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=789503"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=789503"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=789503"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=789503"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=789503"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=789503"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=789503"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=789503"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=789503"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=789503"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}