{"id":1052988,"date":"2024-07-10T18:31:31","date_gmt":"2024-07-11T01:31:31","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-project&p=1052988"},"modified":"2025-03-19T02:21:03","modified_gmt":"2025-03-19T09:21:03","slug":"ar-videos","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/ar-videos\/","title":{"rendered":"Autoregressive Video Models"},"content":{"rendered":"
\n\t
\n\t\t
\n\t\t\t\t\t<\/div>\n\t\t\n\t\t
\n\t\t\t\n\t\t\t
\n\t\t\t\t\n\t\t\t\t
\n\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\n

Autoregressive Video Models<\/h1>\n\n\n\n

Driving large video models with next token prediction<\/p>\n\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t<\/div>\n<\/section>\n\n\n\n\n\n

Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators<\/h2>\n\n\n\n
\n
Paper<\/a><\/div>\n<\/div>\n\n\n\n

In-context learning for vision data has been underexplored compared with that in natural language. Previous works studied image in-context learning, urging models to generate a single image guided by demonstrations. In this project, we propose and study video in-context learning, where the model starts from an existing video clip and generates diverse potential future sequences, each semantically guided by the prompted video demonstrations. To achieve this, we provide a clear definition of the task, and train an autoregressive Transformer on video datasets. We thoroughly analyze the effect of different datasets and represent frames as discrete tokens, and then model them by next token predictions.<\/p>\n\n\n\n

As a result, the obtained vision Transformer model is able to generate a subsequent video sequence of a query video clip, which is semantically aligned with the demonstration video. Demonstration videos are highly versatile and capable of conveying a wide range of information, such as examples for various tasks including moving or grabbing objects, or movements of the camera in an ego-centric video. This allows video in-context learning to address multiple downstream tasks, such as embodied planning and simulating, by letting a query robot imitate the actions demonstrated by other robots, as shown below. Video in-context learning serves as a new and crucial interface for models to interact with the real world<\/em><\/strong>, as videos are good at describing low-level details (where language may fall short) and temporal dynamics (where images are insufficient).<\/p>\n\n\n\n

\"Model
The pipeline of Vid-ICL. Left: Training of Vid-ICL. The data used for training are continuous video clips and the Transformer is trained by next token prediction. Right: In-context Inference of Vid-ICL. The model is conditioned on demonstration videos and generates future frames.<\/figcaption><\/figure>\n\n\n\n

<\/p>\n\n\n\n

Generated Samples on Something-Something v2<\/h4>\n\n\n\n

<\/p>\n\n\n\n

\n
\n

Demonstration<\/p>\n<\/div>\n\n\n\n

\n

DeLVM<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (700M, pt)<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (1.1B, pt)<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (700M, ft)<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (1.1B, ft)<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
\n
\n
\n
\n
\n
\n
\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n<\/figure>\n\n\n\n

Qualitative results on the generated samples by Vid-ICL on Something-Something v2 dataset, a dataset of real world human activities. . The first column shows the demonstration video. The second column shows the generated samples by DeLVM. The third to sixth columns show the generated samples by Vid-ICL with different pretraining and finetuning strategies. It shows that Vid-ICL generates more diverse and plausible samples than DeLVM, and the quality of the generated samples is improved by scaling model sizes and in-domain fine-tuning.<\/p>\n<\/div>\n<\/div>\n\n\n\n

Generated Samples on Robotics Transformer Dataset<\/h4>\n<\/div>\n<\/div>\n\n\n\n
\n
\n

Demonstration<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (700M, pt)<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (1.1B, pt)<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (300M, ft)<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (700M, ft)<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (1.1B, ft)<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n<\/figure>\n\n\n\n

Qualitative results on the generated samples by Vid-ICL on Robotics Transformer(RT-1) dataset, a dataset of robotic manipulation tasks. Vid-ICL offers more concise control over various robotic tasks with the demonstration video.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

Contrastive Demonstration<\/h4>\n\n\n\n

<\/p>\n\n\n\n

\n
\n

Demonstration<\/p>\n<\/div>\n\n\n\n

\n

Generation<\/p>\n<\/div>\n\n\n\n

\n

Contrastive Demonstration<\/p>\n<\/div>\n\n\n\n

\n

Contrastive Generation<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n<\/figure>\n\n\n\n

In this section, we give shared prompts to Vid-ICL in the second and fourth columns of each row, and a contrastive prompt in the first and third columns. Results shows an important feature of Vid-ICL that when conditioned on demonstration with contrastive semantics, Vid-ICL generates contrary samples given the same demonstration, which demonstrates Vid-ICL’s ability to precisely understand the demonstration dynamics and generate diverse future sequences.<\/strong><\/p>\n\n\n\n

<\/p>\n\n\n\n

Text Conditioned Generation<\/h4>\n\n\n\n

<\/p>\n\n\n\n

\n
\n

Text<\/p>\n<\/div>\n\n\n\n

\n

Demonstration<\/p>\n<\/div>\n\n\n\n

\n

Generation w\/o Text<\/p>\n<\/div>\n\n\n\n

\n

Generation w\/ Text<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
\n

Push [something] from left to right<\/p>\n<\/div>\n\n\n\n

\n
\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n<\/figure>\n<\/div>\n<\/div>\n\n\n\n
\n
\n

Turning the camera left while filming [something]<\/p>\n<\/div>\n\n\n\n

\n
\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n<\/figure>\n<\/div>\n<\/div>\n\n\n\n
\n
\n

Moving [something] closer to [something]<\/p>\n<\/div>\n\n\n\n

\n
\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n<\/figure>\n<\/div>\n<\/div>\n\n\n\n

We show that text can be used as an additional in-context condition signals to augment the generation of Vid-ICL. After aligning Vid-ICL to the text, the generated samples are more consistent with the text instructions and demonstration videos.<\/p>\n\n\n\n

<\/p>\n\n\n\n

On Reinforcement Learning Tasks<\/h4>\n\n\n\n
\"gif<\/figure>\n\n\n\n

We demonstrate that Vid-ICL can also function as a simulator in reinforcement learning tasks by evaluating it on the RoboDesk, a test benchmark containing several reinforcement learning tasks. Vid-ICL generates future frames with videos that accomplishing the same task as the demonstration, where the generated frames inversely reflect the corresponding actions that correctly interact with the environment. Evaluating on the Push_red<\/strong><\/em> task, we find that Vid-ICL provides more precise control over the environment interaction.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Ethics Statement<\/h4>\n\n\n\n

Vid-ICL is exclusively a research initiative with no current plans for product integration or public access. We are committed to adhering to Microsoft AI principles during the ongoing development of our models. The datasets utilized in this study are publicly available and have been thoroughly reviewed to ensure they do not include personally identifiable information or offensive content. Nonetheless, as these datasets are sourced from the Internet, there may still be inherent biases. To address this, we have implemented a rigorous filtering process on the training data to minimize the potential for the model to generate inappropriate content.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Citation<\/h4>\n\n\n\n
@article{zhang2024video,\n  title={Video In-Context Learning},\n  author={Zhang, Wentao and Guo, Junliang and He, Tianyu and Zhao, Li and Xu, Linli and Bian, Jiang},\n  journal={arXiv preprint arXiv:2407.07356},\n  year={2024}\n}<\/code><\/pre>\n\n\n\n\n\n
Fast Autoregressive Video Generation with Diagonal Decoding<\/h2>\n\n\n\n
\n
Paper<\/a><\/div>\n\n\n\n
Code (Coming Soon)<\/a><\/div>\n<\/div>\n\n\n\n
Autoregressive Transformer models have demonstrated impressive performance in video generation, but their sequential token-by-token decoding process poses a major bottleneck, particularly for long videos represented by tens of thousands of tokens. Here, we propose Diagonal Decoding (DiagD)<\/strong>, a training-free inference acceleration algorithm for autoregressively pre-trained models that exploits spatial and temporal correlations in videos. Our method generates tokens along diagonal paths in the spatial-temporal token grid, enabling parallel decoding within each frame as well as partially overlapping across consecutive frames. The proposed algorithm is versatile and adaptive to various generative models and tasks, while providing flexible control over the trade-off between inference speed and visual quality. Experiments on multiple autoregressive video generation models and datasets demonstrate that DiagD achieves up to 10x speedup compared to naive sequential decoding<\/strong>, while maintaining comparable visual fidelity.<\/p>\n\n\n\n
\"Comparisons<\/figure>\n\n\n\n
\n\n\n\n
Method<\/h4>\n\n\n\n
\"Illustration<\/figure>\n\n\n\n
The motivation of our method arises from intuitive observations on consecutive frames in a video, which can be summarized in two key insights. As shown above, the first insight is that patches exhibit stronger correlations with their spatial neighbors than with sequential ones. Secondly, due to the temporal redundancy of videos, patches from consecutive frames that occupy similar relative positions are highly likely to be similar to each other. <\/p>\n\n\n\n
As a result, we propose Diagonal Decoding, an iterative algorithm that generates tokens along diagonal paths in the spatial-temporal token grid. Spatially, within each frame, tokens along the same diagonal are generated in parallel, leveraging the strong local dependencies between neighboring patches. And temporally, by stacking frames together, our method generates the top-left tokens of the next frame before completing the current frame, as these tokens are less likely to depend on the bottom-right tokens that have not yet been generated.<\/p>\n\n\n\n
\n\n\n\n
Case: Cosmos 12B Autoregressive Model<\/h4>\n\n\n\n
\n
\n
Next Token Prediction<\/p>\n<\/div>\n\n\n\n
\n
DiagD (k=1)<\/p>\n<\/div>\n\n\n\n
\n
DiagD (k=2)<\/p>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\"NTP<\/figure>\n<\/div>\n\n\n\n
\n
\"Diagd<\/figure>\n<\/div>\n\n\n\n
\n
\"Diagd<\/figure>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\"NTP<\/figure>\n<\/div>\n\n\n\n
\n
\"Diagd<\/figure>\n<\/div>\n\n\n\n
\n
\"Diagd<\/figure>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\"NTP<\/figure>\n<\/div>\n\n\n\n
\n
\"Diagd<\/figure>\n<\/div>\n\n\n\n
\n
\"Diagd<\/figure>\n<\/div>\n<\/div>\n\n\n\n
\n\n\n\n
Case: WHAM 1.6B Model<\/h4>\n\n\n\n
\n
\n
Next Token Prediction<\/p>\n<\/div>\n\n\n\n
\n
DiagD (k=1)<\/p>\n<\/div>\n\n\n\n
\n
DiagD (k=2)<\/p>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\"NTP<\/figure>\n<\/div>\n\n\n\n
\n
\"Diagd<\/figure>\n<\/div>\n\n\n\n
\n
\"Diagd<\/figure>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\"NTP<\/figure>\n<\/div>\n\n\n\n
\n
\"Diagd<\/figure>\n<\/div>\n\n\n\n
\n
\"Diagd<\/figure>\n<\/div>\n<\/div>\n\n\n\n
\n\n\n\n
Case: Autoregressive Model on Minecraft<\/h4>\n\n\n\n
\n
\n
Next Token Prediction<\/p>\n<\/div>\n\n\n\n
\n
DiagD w\/o Finetune<\/p>\n<\/div>\n\n\n\n
\n
DiagD w\/ Finetune<\/p>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\"a<\/figure>\n<\/div>\n\n\n\n
\n
\"a<\/figure>\n<\/div>\n\n\n\n
\n
\"a<\/figure>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\"a<\/figure>\n<\/div>\n\n\n\n
\n
\"a<\/figure>\n<\/div>\n\n\n\n
\n
\"a<\/figure>\n<\/div>\n<\/div>\n\n\n\n
\n\n\n\n
BibTex<\/h4>\n\n\n\n
@article{ye2025fast,\n      title={Fast Autoregressive Video Generation with Diagonal Decoding}, \n      author={Ye, Yang and Guo, Junliang and Wu, Haoyu and He, Tianyu and Pearce, Tim and Rashid, Tabish and Hofmann, Katja and Bian, Jiang},\n      journal={arXiv preprint arXiv:2503.14070},\n      year={2025},\n}<\/code><\/pre>\n\n\n","protected":false},"excerpt":{"rendered":"
Driving large video models with next token prediction In-context learning for vision data has been underexplored compared with that in natural language. Previous works studied image in-context learning, urging models to generate a single image guided by demonstrations. In this project, we propose and study video in-context learning, where the model starts from an existing […]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"research-area":[13556,13562],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-1052988","msr-project","type-msr-project","status-publish","hentry","msr-research-area-artificial-intelligence","msr-research-area-computer-vision","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"","related-publications":[1056966],"related-downloads":[],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[],"slides":[],"related-researchers":[{"type":"user_nicename","display_name":"Li Zhao","user_id":36152,"people_section":"Related people","alias":"lizo"}],"msr_research_lab":[199560],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/1052988","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":50,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/1052988\/revisions"}],"predecessor-version":[{"id":1140261,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/1052988\/revisions\/1140261"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1052988"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1052988"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1052988"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1052988"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=1052988"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}