Furu Wei

Distinguished Scientist, Chief Scientist of Microsoft Research Asia

Publications

View by:

- DocReward: A Document Reward Model for Structuring and Stylizing
  
  Junpeng Liu, Yuzhong Zhao, Bowen Cao, Jiayu Ding, Yilin Jia, Tengchao Lv, Yupan Huang, Shaohan Huang, Nan Yang, Li Dong, Lei Cui, Tao Ge, Xun Wang, Huitian Jiao, Sun Mao, Fnu Kartik, Siqing Chen, Wai Lam, Furu Wei
  
  October 2025
  
  Publication
- Chain-of-Retrieval Augmented Generation
  
  Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, Furu Wei
  
  NeurIPS 2025 | October 2025
  
  Publication
- Native Hybrid Thinking Models
  
  Lingjie Jiang, Xun Wu, Shaohan Huang, Qingxiu Dong, Zewen Chi, Li Dong, Xingxing Zhang, Tengchao Lv, Lei Cui, Furu Wei
  
  NeurIPS 2025 | October 2025
  
  Publication
- Scaling Laws of Synthetic Data for Language Models
  
  Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R. Fung, Weizhu Chen, Minhao Cheng, Furu Wei
  
  COLM 2025 | July 2025
  
  Publication
- Reward Reasoning Model
  
  Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, Furu Wei
  
  NeurIPS 2025 | May 2025
  
  DOI Publication Publication
- Chain-of-Reasoning: Towards Unified Mathematical Reasoning in Large Language Models via a Multi-Paradigm Perspective
  
  Yiyao Yu, Yuxiang Zhang, Dongdong Zhang, Xiao Liang, Hengyuan Zhang, Xingxing Zhang, Mahmoud Khademi, Hany Awadalla, Junjie Wang, Yujiu Yang, Furu Wei
  
  2025 Meeting of the Association for Computational Linguistics | May 2025
  
  Publication Github
- Boosting Large Language Model for Speech Synthesis: An Empirical Study
  
  Hongkun Hao, Long Zhou, Shujie Liu, Jinyu Li, Shujie Hu, Rui Wang, Furu Wei
  
  ICASSP | April 2025
  
  Publication
- Model as a Game: On Numerical and Spatial Consistency for Generative Games
  
  Jingye Chen, Yuzhong Zhao, Yupan Huang, Lei Cui, Li Dong, Tengchao Lv, Qifeng Chen, Furu Wei
  
  March 2025
  
  March 2025
  
  Publication
- Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning
  
  Wenkai Yang, Shuming Ma, Yankai Lin, Furu Wei
  
  NeurIPS 2025 | February 2025
  
  DOI Publication Publication
- Examining False Positives under Inference Scaling for Mathematical Reasoning
  
  Yu Wang, Nan Yang, Liang Wang, Furu Wei
  
  MSR-TR-2025-2 | February 2025
  
  Published by Microsoft
  
  Publication
- PEACE: Empowering Geologic Map Holistic Understanding with MLLMs
  
  Yangyu Huang, Tianyi Gao, Haoran Xu, Qihao Zhao, Yang Song, Zhipeng Gui, Tengchao Lv, Hao Cheng, Lei Cui, Scarlett Li, Furu Wei
  
  January 2025
  
  Publication Access
- Imagine while Reasoning in Space: Multimodal Visualization-of-Thought
  
  Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vuli'c, Furu Wei
  
  ICML 2025 | January 2025
  
  DOI Publication Publication
- Bootstrap Your Own Context Length
  
  Liang Wang, Nan Yang, Xingxing Zhang, Xiaolong Huang, Furu Wei
  
  ArXiv | December 2024, Vol abs/2412.18860
  
  DOI Publication Publication
- REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models
  
  Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang
  
  December 2024
  
  Publication
- MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark
  
  Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Furu Wei, Qinzheng Sun, Ying Xin, Shaoguang Mao, Xin Zhang, Qiufeng Yin, Scarlett Li
  
  December 2024
  
  Publication Github
- BitNet a4.8: 4-bit Activations for 1-bit LLMs
  
  Hongyu Wang, Shuming Ma, Furu Wei
  
  November 2024
  
  Work in progress
  
  Publication Access
- Little Giants: Synthesizing High-Quality Embedding Data at Scale
  
  Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, Zhicheng Dou
  
  NAACL 2025 | October 2024
  
  DOI Publication Publication
- 1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs
  
  Jinheng Wang, Hansong Zhou, Ting Song, Shaoguang Mao, Shuming Ma, Hongyu Wang, Yan Xia, Furu Wei
  
  October 2024
  
  Publication
- Differential Transformer
  
  Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, Furu Wei
  
  MSR-TR-2024-42 | October 2024
  
  Published by Microsoft
  
  Publication
- ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation
  
  Zongyi Li, Shujie Hu, Shujie Liu, Long Zhou, Jeongsoo Choi, Lingwei Meng, Xun Guo, Jinyu Li, Hefei Ling, Furu Wei
  
  October 2024
  
  October 2024
  
  Publication
- Boosting Text-to-Video Generative Model with MLLMs Feedback
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
  
  Hongyu Wang, Shuming Ma, Ruiping Wang, Furu Wei
  
  July 2024
  
  Publication
- Autoregressive Speech Synthesis without Vector Quantization
  
  Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, Furu Wei
  
  July 2024
  
  Publication Project
- VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
  
  Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, Furu Wei
  
  June 2024
  
  Publication Project
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
  
  Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
  
  June 2024
  
  Publication Project
- Direct Preference Knowledge Distillation for Large Language Models
  
  Yixing Li, Yuxian Gu, Li Dong, Dequan Wang, Yu Cheng, Furu Wei
  
  June 2024
  
  DOI Publication Publication
- Instruction Pre-Training: Language Models are Supervised Multitask Learners
  
  Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei
  
  June 2024
  
  DOI Publication Publication
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
  
  Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
  
  June 2024
  
  DOI Publication Publication Project
- A whole-slide foundation model for digital pathology from real-world data
  
  Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, Yanbo Xu, Mu-Hsin Wei, Wenhui Wang, Shuming Ma, Furu Wei, Jianwei Yang, Chunyuan Li, Jianfeng Gao, Jaylen Rosemon, Tucker Bower, Soohee Lee, R. Weerasinghe, Bill Wright, Ari Robicsek, Brian Piening, Carlo Bifulco, Sheng Wang, Hoifung Poon
  
  Nature | May 2024
  
  Publication DOI Publication
- xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
  
  Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, Dongyan Zhao
  
  NeurIPS 2024 | May 2024
  
  DOI Publication Publication
- MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
  
  Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
  
  May 2024
  
  DOI Publication Publication
- You Only Cache Once: Decoder-Decoder Architectures for Language Models
  
  Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei
  
  NeurIPS 2024 | May 2024
  
  DOI Publication Publication
- Kosmos-G: Generating Images in Context with Multimodal Large Language Models
  
  Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei
  
  ICLR 2024 | May 2024
  
  DOI Publication PDF
- Multi-Head Mixture-of-Experts
  
  Xun Wu, Shaohan Huang, Wenhui Wang, Furu Wei
  
  NeurIPS 2024 | April 2024
  
  DOI Publication Publication
- LongEmbed: Extending Embedding Models for Long Context Retrieval
  
  Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
  
  EMNLP 2024 | April 2024
  
  DOI Publication Publication
- Mind’s Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models
  
  Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, Furu Wei
  
  NeurIPS 2024 | April 2024
  
  Publication
- Mixture of LoRA Experts
  
  Shaohan Huang, Furu Wei
  
  ICLR 2024 | April 2024
  
  Publication
- LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models
  
  Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, Furu Wei
  
  March 2024
  
  DOI Publication Publication
- WavLLM: Towards Robust and Adaptive Speech Large Language Model
  
  Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei
  
  March 2024
  
  DOI Publication Publication
- SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data
  
  Ziqiang Zhang, Sanyuan Chen, Long Zhou, Yu Wu, Shuo Ren, Shujie Liu, Zhuoyuan Yao, Xun Gong, Lirong Dai, Jinyu Li, Furu Wei
  
  IEEE/ACM Transactions on Audio, Speech, and Language Processing | March 2024
  
  Publication
- MathScale: Scaling Instruction Tuning for Mathematical Reasoning
  
  Zhengyang Tang, Xingxing Zhang, Benyou Wang, Furu Wei
  
  ICML 2024 | March 2024
  
  DOI Publication Publication
- ResLoRA: Identity Residual Mapping in Low-Rank Adaption
  
  Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  February 2024
  
  DOI Publication Publication
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
  
  Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- Towards Optimal Learning of Language Models
  
  Yuxian Gu, Li Dong, Yaru Hao, Qingxiu Dong, Minlie Huang, Furu Wei
  
  February 2024
  
  DOI Publication Publication
- Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models
  
  Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, Ji-Rong Wen
  
  ACL 2024 | February 2024
  
  DOI Publication Publication
- Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
  
  Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- Generative Representational Instruction Tuning
  
  Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, Douwe Kiela
  
  ICLR 2025 | February 2024
  
  DOI Publication Publication
- Multilingual E5 Text Embeddings: A Technical Report
  
  Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
  
  MSR-TR-2024-45 | February 2024
  
  Published by Microsoft
  
  DOI Publication Publication Github
- Text Diffusion with Reinforced Conditioning
  
  Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  AAAI 2024 | February 2024
  
  Publication
- Improving Text Embeddings with Large Language Models
  
  Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
  
  ACL 2024 | January 2024
  
  Preprint
- When an Image is Worth 1,024 x 1,024 Words: A Case Study in Computational Pathology
  
  Wenhui Wang, Shuming Ma, Hanwen Xu, Naoto Usuyama, Jiayu Ding, Hoifung Poon, Furu Wei
  
  December 2023
  
  DOI Publication Publication
- Large Search Model: Redefining Search Stack in the Era of LLMs
  
  Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
  
  SIGIR Forum | December 2023, Vol 57(2)
  
  Publication
- BitNet: Scaling 1-bit Transformers for Large Language Models
  
  Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei
  
  October 2023
  
  DOI Publication
- Fine-Tuning LLaMA for Multi-Stage Text Retrieval
  
  Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, Jimmy Lin
  
  SIGIR 2024 | October 2023
  
  PDF
- Augmenting Language Models with Long-Term Memory
  
  Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, Furu Wei
  
  NeurIPS 2023 | October 2023
  
  Publication
- PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
  
  Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
  
  ICLR 2024 | October 2023
  
  Publication
- Adapting Large Language Models via Reading Comprehension
  
  Daixuan Cheng, Shaohan Huang, Furu Wei
  
  ICLR 2024 | September 2023
  
  DOI
- Kosmos-2.5: A Multimodal Literate Model
  
  Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
  
  September 2023
  
  Preprint Project
- Retentive Network: A Successor to Transformer for Large Language Models
  
  Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei
  
  August 2023
  
  Publication
- Generative Retrieval for Conversational Question Answering
  
  Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
  
  Information Processing and Management | August 2023, Vol 60(5)
  
  Publication
- LongNet: Scaling Transformers to 1,000,000,000 Tokens
  
  Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei
  
  July 2023
  
  DOI PDF Video
- In-context Autoencoder for Context Compression in a Large Language Model
  
  Tao Ge, Jing Hu, Haixun Wang, Si-Qing Chen, Furu Wei
  
  ICLR 2024 | July 2023
  
  DOI Publication
- Learning to Retrieve In-Context Examples for Large Language Models
  
  Liang Wang, Nan Yang, Furu Wei
  
  EACL 2024 | July 2023
  
  Publication
- Knowledge Distillation of Large Language Models
  
  Yuxian Gu, Li Dong, Furu Wei, Minlie Huang
  
  ICLR 2024 | June 2023
  
  DOI PDF
- Learning to Rank in Generative Retrieval
  
  Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
  
  AAAI 2024 | June 2023
  
  Publication
- Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting
  
  Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, Furu Wei
  
  2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) | May 2023, pp. 12365-12394
  
  DOI Publication Publication Publication
- Multiview Identifiers Enhanced Generative Retrieval
  
  Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
  
  ACL 2023 | May 2023
  
  Publication
- TextDiffuser: Diffusion Models as Text Painters
  
  Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei
  
  NeurIPS 2023 | May 2023
  
  Publication
- VatLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
  
  Qiushi Zhu, Long Zhou, Ziqiang Zhang, Shujie Liu, Binxing Jiao, Jie Zhang, Lirong Dai, Daxin Jiang, Jinyu Li, Furu Wei
  
  IEEE Transactions on Multimedia | May 2023
  
  Publication
- Inference with Reference: Lossless Acceleration of Large Language Models
  
  Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang (姜大昕), Linjun Yang, Rangan Majumder, Furu Wei
  
  MSR-TR-2023-17 | April 2023
  
  Published by Microsoft
  
  Publication
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- Query2doc: Query Expansion with Large Language Models
  
  Liang Wang, Nan Yang, Furu Wei
  
  EMNLP 2023 | March 2023
  
  Publication
- TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
  
  Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei
  
  AAAI 2023 | February 2023
  
  Publication Project
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
  
  Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, Furu Wei
  
  January 2023
  
  Publication PDF Project
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
  
  Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang (姜大昕), Rangan Majumder, Furu Wei
  
  December 2022
  
  Publication
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- DiT: Self-supervised Pre-training for Document Image Transformer
  
  Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei
  
  ACM Multimedia 2022 | October 2022
  
  Publication Project
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
  
  Yupang Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei
  
  ACM Multimedia 2022 | October 2022
  
  Publication Project
- SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
  
  Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, Furu Wei
  
  EMNLP 2022 | October 2022
  
  Publication
- Separating Long-Form Speech with Group-Wise Permutation Invariant Training
  
  Wangyou Zhang, Zhuo Chen, Naoyuki Kanda, Shujie Liu, Jinyu Li, Sefik Emre Eskimez, Takuya Yoshioka, Xiong Xiao, Zhong Meng, Yanmin Qian, Furu Wei
  
  Interspeech 2022 | September 2022
  
  Publication
- Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
  
  Wenhui Wang, Hangbo Bao, Li Dong, Kriti Aggarwal, Saksham Singhal, Subhojit Som, Furu Wei, Johan Bjorck, Zhiliang Peng, Qiang Liu, Owais Khan Mohammed
  
  August 2022
  
  Publication
- Learning Diverse Document Representations with Deep Query Interactions for Dense Retrieval
  
  Zehan Li, Nan Yang, Liang Wang, Furu Wei
  
  August 2022
  
  Publication
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
  
  Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, Furu Wei
  
  IEEE Journal of Selected Topics in Signal Processing | July 2022, Vol 16(6): pp. 1505-1518
  
  Publication
- SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
  
  Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang (姜大昕), Rangan Majumder, Furu Wei
  
  ACL 2023 | July 2022
  
  Publication
- Lossless Acceleration for Seq2seq Generation with Aggressive Decoding
  
  Tao Ge, Heming Xia, Xin Sun, Si-Qing Chen, Furu Wei
  
  May 2022
  
  Publication
- UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
  
  Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, Xiangzhan Yu
  
  ICASSP 2022 | May 2022
  
  Publication
- Attention Temperature Matters in Abstractive Summarization Distillation
  
  Shengqiang Zhang, Xingxing Zhang, Hangbo Bao, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment
  
  Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- Controllable Natural Language Generation with Contrastive Prefixes
  
  Jing Qian, Li Dong, Yelong Shen, Furu Wei, Weizhu Chen
  
  ACL 2022 | May 2022
  
  Publication
- Knowledge Neurons in Pretrained Transformers
  
  Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding
  
  Junlong Li, Yiheng Xu, Lei Cui, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication Project
- Neural Label Search for Zero-Shot Multi-Lingual Extractive Summarization
  
  Ruipeng Jia, Xingxing Zhang, Yanan Cao, Shi Wang, Zheng Lin, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
  
  Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- StableMoE: Stable Routing Strategy for Mixture of Experts
  
  Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
  
  Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang (姜大昕), Haoyi Zhou, Jianxin Li, Furu Wei
  
  May 2022
- Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
  
  Guanhua Chen, Shuming Ma, Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding
  
  Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication Project
- BEiT: BERT Pre-Training of Image Transformers
  
  Hangbo Bao, Li Dong, Songhao Piao, Furu Wei
  
  ICLR 2022 | April 2022
  
  Publication
- Sequence Level Contrastive Learning for Text Summarization
  
  Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei
  
  AAAI 2022 | January 2022
  
  Publication
- Document AI: Benchmarks, Models and Applications
  
  Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
  
  November 2021
  
  Publication Project
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication Project
- LayoutReader: Pre-training of Text and Layout for Reading Order Detection
  
  Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, Furu Wei
  
  EMNLP 2021 | November 2021
  
  Publication Project
- MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
  
  Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  EMNLP 2021 | October 2021
  
  Publication
- Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
  
  Guanhua Chen, Shuming Ma, Yun Chen, Li Dong, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei
  
  EMNLP 2021 | October 2021
  
  Publication
- Jointly Learning to Repair Code and Generate Commit Message
  
  Jiaqi Bai, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Zhoujun Li
  
  2021 Empirical Methods in Natural Language Processing | September 2021
  
  Publication
- Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
  
  Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei
  
  2021 Empirical Methods in Natural Language Processing | September 2021
  
  Publication
- Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
  
  Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian McAuley, Furu Wei
  
  2021 Empirical Methods in Natural Language Processing | September 2021
  
  Publication
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication Project
- Document AI: Benchmarks, Models and Applications (In Chinese)
  
  Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
  
  CCL 2021 | August 2021
  
  Project
- UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
  
  Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang
  
  2021 International Conference on Machine Learning | July 2021
  
  Publication
- Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
  
  Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | July 2021
  
  Publication Github
- Grammar-Based Patches Generation for Automated Program Repair
  
  Yu Tang, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Muyun Yang
  
  ACL-IJCNLP 2021 | July 2021
  
  PDF
- Multilingual Agreement for Multilingual Neural Machine Translation
  
  Jian Yang, Yuwei Yin, Shuming Ma, Haoyang Huang, Dongdong Zhang, Zhoujun Li, Furu Wei
  
  July 2021
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication Project
- Learning to Sample Replacements for ELECTRA Pre-Training
  
  Yaru Hao, Li Dong, Hangbo Bao, Ke Xu, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication Project
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication Project
- Consistency Regularization for Cross-Lingual Fine-Tuning
  
  Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
  
  Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
  
  Xin Sun, Tao Ge, Furu Wei, Houfeng Wang
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization
  
  Tengchao Lv, Lei Cui, Momcilo Vasilijevic, Furu Wei
  
  June 2021
  
  Preprint
- Memory-Efficient Differentiable Transformer Architecture Search
  
  Yuekai Zhao, Li Dong, Yelong Shen, Zhihua Zhang, Furu Wei, Wei Chen
  
  ACL-IJCNLP 2021 | May 2021
  
  Publication
- Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
  
  Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian J. McAuley, Furu Wei
  
  April 2021
  
  Publication
- LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
  
  Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
  
  April 2021
  
  Publication Project
- LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
  
  Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou
  
  ACL-IJCNLP 2021 | January 2021
  
  Preprint Publication Github Project
- MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
  
  Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | December 2020
  
  PDF Publication
- XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.
  
  Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko I. Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei
  
  December 2020
  
  Publication Publication Publication Project
- DocBank: A Benchmark Dataset for Document Layout Analysis
  
  Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, Ming Zhou
  
  COLING 2020 | December 2020
  
  Publication Project
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
  
  Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao
  
  ECCV | August 2020
  
  Publication Video Github Project Project Project
- LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  
  Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
  
  KDD 2020 | August 2020
  
  PDF Publication Github Project
- UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
  
  Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
  
  37th International Conference on Machine Learning (ICML 2020) | July 2020
  
  Publication Github Project
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
  
  Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
  
  July 2020
  
  Publication Publication Github
- Multimodal Matching Transformer for Live Commenting
  
  Chaoqun Duan, Lei Cui, Shuming Ma, Furu Wei, Conghui Zhu, Tiejun Zhao
  
  ECAI 2020 | July 2020
  
  Publication
- TableBank: Table Benchmark for Image-based Table Detection and Recognition
  
  Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, Zhoujun Li
  
  LREC 2020 | May 2020
  
  Publication Project
- Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
  
  Yaru Hao, Li Dong, Furu Wei, Ke Xu
  
  AAAI 2021 | April 2020
  
  AAAI 2021 Best Paper Runner Up
  
  Publication PDF
- Self-Adversarial Learning with Comparative Discrimination for Text Generation
  
  Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, Ming Zhou
  
  Eighth International Conference on Learning Representations (ICLR) | April 2020
  
  Publication PDF
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
  
  Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai
  
  Eighth International Conference on Learning Representations (ICLR) | April 2020
  
  Publication PDF Github
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
  
  Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, Ming Zhou
  
  NeurIPS 2020 | February 2020
  
  Publication Project
- Unified Language Model Pre-training for Natural Language Understanding and Generation
  
  Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
  
  33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019
  
  Publication Github Project
- Neural Melody Composition from Lyrics
  
  Hangbo Bao, Shaohan Huang, Furu Wei, Lei Cui, Yu Wu, Chuanqi Tan, Songhao Piao, Ming Zhou
  
  NLPCC 2019 | September 2019
  
  Publication
- LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
  
  Shuming Ma, Lei Cui, Damai Dai, Furu Wei, Xu Sun
  
  AAAI 2019 | July 2019
  
  Publication Publication
- Retrieval-Enhanced Adversarial Training for Neural Response Generation
  
  Qingfu Zhu, Lei Cui, Wei-Nan Zhang, Furu Wei, Ting Liu
  
  ACL 2019 | July 2019
  
  Publication
- Fine-grained Coordinated Cross-lingual Text Stream Alignment for Endless Language Knowledge Acquisition
  
  Tao Ge, Qing Dou, Heng Ji, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
  
  EMNLP 2018 | November 2018
- SeRI: A Dataset for Sub-event Relation Inference from an Encyclopedia
  
  Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
  
  NLPCC 2018 | August 2018
  
  Publication
- Neural Open Information Extraction
  
  Lei Cui, Furu Wei, Ming Zhou
  
  ACL 2018 | July 2018
  
  Publication Publication
- Attention-Fused Deep Matching Network for Natural Language Inference
  
  Chaoqun Duan, Lei Cui, Xinchi Chen, Furu Wei, Conghui Zhu, Tiejun Zhao
  
  IJCAI 2018 | July 2018
  
  Publication
- EventWiki: A Knowledge Base of Major Events
  
  Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
  
  LREC 2018 | May 2018
  
  Publication Publication Publication
- Leveraging Contextual Sentence Relations for Extractive Summarization Using a Neural Attention Model
  
  Pengjie Ren, Zhumin Chen, Zhaochun Ren, Furu Wei, Jun Ma, Maarten de Rijke
  
  SIGIR '17, Shinjuku, Tokyo, Japan, August 07-11, 2017 | August 2017
  
  Publication
- SuperAgent: A Customer Service Chatbot for E-commerce Websites
  
  Lei Cui, Furu Wei, Shaohan Huang, Chuanqi Tan, Chaoqun Duan, Ming Zhou
  
  ACL 2017 | July 2017
  
  Publication Publication
- QuickView: advanced search of tweets
  
  Xiaohua Liu, Long Jiang, Furu Wei, Ming Zhou, QuickView Team Microsoft
  
  SIGIR (Demo paper) | January 2011
  
  ISBN: 978-1-4503-0757-4
  
  Publication Publication
- OpinionSeer: Interactive Visualization of Hotel Customer Feedback
  
  Yingcai Wu, Furu Wei, Shixia Liu, Norman Au, Weiwei Cui, Hong Zhou, Huamin Qu
  
  IEEE Transactions on Visualization and Computer Graphics | November 2010, pp. 1109-1118
  
  Publication Publication
- Context-Preserving, Dynamic Word Cloud Visualization
  
  Weiwei Cui, Yingcai Wu, Shixia Liu, Furu Wei, Michelle Zhou, Huamin Qu
  
  IEEE Computer Graphics and Applications | September 2010, Vol 30(6): pp. 42
  
  Publication Publication
- Context Preserving Dynamic Word Cloud Visualization
  
  Weiwei Cui, Yingcai Wu, Shixia Liu, Furu Wei, Michelle Zhou, Huamin Qu
  
  2010 IEEE Pacific Visualization Symposium (PacificVis) | March 2010
  
  Publication Publication

- DocReward: A Document Reward Model for Structuring and Stylizing
  
  Junpeng Liu, Yuzhong Zhao, Bowen Cao, Jiayu Ding, Yilin Jia, Tengchao Lv, Yupan Huang, Shaohan Huang, Nan Yang, Li Dong, Lei Cui, Tao Ge, Xun Wang, Huitian Jiao, Sun Mao, Fnu Kartik, Siqing Chen, Wai Lam, Furu Wei
  
  October 2025
  
  Publication
- Native Hybrid Thinking Models
  
  Lingjie Jiang, Xun Wu, Shaohan Huang, Qingxiu Dong, Zewen Chi, Li Dong, Xingxing Zhang, Tengchao Lv, Lei Cui, Furu Wei
  
  NeurIPS 2025 | October 2025
  
  Publication
- Scaling Laws of Synthetic Data for Language Models
  
  Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R. Fung, Weizhu Chen, Minhao Cheng, Furu Wei
  
  COLM 2025 | July 2025
  
  Publication
- Reward Reasoning Model
  
  Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, Furu Wei
  
  NeurIPS 2025 | May 2025
  
  DOI Publication Publication
- Chain-of-Reasoning: Towards Unified Mathematical Reasoning in Large Language Models via a Multi-Paradigm Perspective
  
  Yiyao Yu, Yuxiang Zhang, Dongdong Zhang, Xiao Liang, Hengyuan Zhang, Xingxing Zhang, Mahmoud Khademi, Hany Awadalla, Junjie Wang, Yujiu Yang, Furu Wei
  
  2025 Meeting of the Association for Computational Linguistics | May 2025
  
  Publication Github
- Model as a Game: On Numerical and Spatial Consistency for Generative Games
  
  Jingye Chen, Yuzhong Zhao, Yupan Huang, Lei Cui, Li Dong, Tengchao Lv, Qifeng Chen, Furu Wei
  
  March 2025
  
  March 2025
  
  Publication
- Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning
  
  Wenkai Yang, Shuming Ma, Yankai Lin, Furu Wei
  
  NeurIPS 2025 | February 2025
  
  DOI Publication Publication
- Imagine while Reasoning in Space: Multimodal Visualization-of-Thought
  
  Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vuli'c, Furu Wei
  
  ICML 2025 | January 2025
  
  DOI Publication Publication
- MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark
  
  Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Furu Wei, Qinzheng Sun, Ying Xin, Shaoguang Mao, Xin Zhang, Qiufeng Yin, Scarlett Li
  
  December 2024
  
  Publication Github
- BitNet a4.8: 4-bit Activations for 1-bit LLMs
  
  Hongyu Wang, Shuming Ma, Furu Wei
  
  November 2024
  
  Work in progress
  
  Publication Access
- 1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs
  
  Jinheng Wang, Hansong Zhou, Ting Song, Shaoguang Mao, Shuming Ma, Hongyu Wang, Yan Xia, Furu Wei
  
  October 2024
  
  Publication
- Differential Transformer
  
  Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, Furu Wei
  
  MSR-TR-2024-42 | October 2024
  
  Published by Microsoft
  
  Publication
- ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation
  
  Zongyi Li, Shujie Hu, Shujie Liu, Long Zhou, Jeongsoo Choi, Lingwei Meng, Xun Guo, Jinyu Li, Hefei Ling, Furu Wei
  
  October 2024
  
  October 2024
  
  Publication
- Boosting Text-to-Video Generative Model with MLLMs Feedback
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
  
  Hongyu Wang, Shuming Ma, Ruiping Wang, Furu Wei
  
  July 2024
  
  Publication
- Autoregressive Speech Synthesis without Vector Quantization
  
  Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, Furu Wei
  
  July 2024
  
  Publication Project
- VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
  
  Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, Furu Wei
  
  June 2024
  
  Publication Project
- Direct Preference Knowledge Distillation for Large Language Models
  
  Yixing Li, Yuxian Gu, Li Dong, Dequan Wang, Yu Cheng, Furu Wei
  
  June 2024
  
  DOI Publication Publication
- Instruction Pre-Training: Language Models are Supervised Multitask Learners
  
  Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei
  
  June 2024
  
  DOI Publication Publication
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
  
  Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
  
  June 2024
  
  DOI Publication Publication Project
- A whole-slide foundation model for digital pathology from real-world data
  
  Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, Yanbo Xu, Mu-Hsin Wei, Wenhui Wang, Shuming Ma, Furu Wei, Jianwei Yang, Chunyuan Li, Jianfeng Gao, Jaylen Rosemon, Tucker Bower, Soohee Lee, R. Weerasinghe, Bill Wright, Ari Robicsek, Brian Piening, Carlo Bifulco, Sheng Wang, Hoifung Poon
  
  Nature | May 2024
  
  Publication DOI Publication
- xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
  
  Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, Dongyan Zhao
  
  NeurIPS 2024 | May 2024
  
  DOI Publication Publication
- MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
  
  Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
  
  May 2024
  
  DOI Publication Publication
- You Only Cache Once: Decoder-Decoder Architectures for Language Models
  
  Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei
  
  NeurIPS 2024 | May 2024
  
  DOI Publication Publication
- Kosmos-G: Generating Images in Context with Multimodal Large Language Models
  
  Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei
  
  ICLR 2024 | May 2024
  
  DOI Publication PDF
- Multi-Head Mixture-of-Experts
  
  Xun Wu, Shaohan Huang, Wenhui Wang, Furu Wei
  
  NeurIPS 2024 | April 2024
  
  DOI Publication Publication
- Mind’s Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models
  
  Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, Furu Wei
  
  NeurIPS 2024 | April 2024
  
  Publication
- LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models
  
  Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, Furu Wei
  
  March 2024
  
  DOI Publication Publication
- WavLLM: Towards Robust and Adaptive Speech Large Language Model
  
  Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei
  
  March 2024
  
  DOI Publication Publication
- MathScale: Scaling Instruction Tuning for Mathematical Reasoning
  
  Zhengyang Tang, Xingxing Zhang, Benyou Wang, Furu Wei
  
  ICML 2024 | March 2024
  
  DOI Publication Publication
- ResLoRA: Identity Residual Mapping in Low-Rank Adaption
  
  Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  February 2024
  
  DOI Publication Publication
- Towards Optimal Learning of Language Models
  
  Yuxian Gu, Li Dong, Yaru Hao, Qingxiu Dong, Minlie Huang, Furu Wei
  
  February 2024
  
  DOI Publication Publication
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
  
  Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models
  
  Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, Ji-Rong Wen
  
  ACL 2024 | February 2024
  
  DOI Publication Publication
- Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
  
  Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- Multilingual E5 Text Embeddings: A Technical Report
  
  Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
  
  MSR-TR-2024-45 | February 2024
  
  Published by Microsoft
  
  DOI Publication Publication Github
- Text Diffusion with Reinforced Conditioning
  
  Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  AAAI 2024 | February 2024
  
  Publication
- When an Image is Worth 1,024 x 1,024 Words: A Case Study in Computational Pathology
  
  Wenhui Wang, Shuming Ma, Hanwen Xu, Naoto Usuyama, Jiayu Ding, Hoifung Poon, Furu Wei
  
  December 2023
  
  DOI Publication Publication
- BitNet: Scaling 1-bit Transformers for Large Language Models
  
  Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei
  
  October 2023
  
  DOI Publication
- Augmenting Language Models with Long-Term Memory
  
  Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, Furu Wei
  
  NeurIPS 2023 | October 2023
  
  Publication
- Adapting Large Language Models via Reading Comprehension
  
  Daixuan Cheng, Shaohan Huang, Furu Wei
  
  ICLR 2024 | September 2023
  
  DOI
- Kosmos-2.5: A Multimodal Literate Model
  
  Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
  
  September 2023
  
  Preprint Project
- Retentive Network: A Successor to Transformer for Large Language Models
  
  Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei
  
  August 2023
  
  Publication
- LongNet: Scaling Transformers to 1,000,000,000 Tokens
  
  Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei
  
  July 2023
  
  DOI PDF Video
- In-context Autoencoder for Context Compression in a Large Language Model
  
  Tao Ge, Jing Hu, Haixun Wang, Si-Qing Chen, Furu Wei
  
  ICLR 2024 | July 2023
  
  DOI Publication
- Knowledge Distillation of Large Language Models
  
  Yuxian Gu, Li Dong, Furu Wei, Minlie Huang
  
  ICLR 2024 | June 2023
  
  DOI PDF
- Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting
  
  Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, Furu Wei
  
  2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) | May 2023, pp. 12365-12394
  
  DOI Publication Publication Publication
- TextDiffuser: Diffusion Models as Text Painters
  
  Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei
  
  NeurIPS 2023 | May 2023
  
  Publication
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
  
  Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei
  
  AAAI 2023 | February 2023
  
  Publication Project
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
  
  Yupang Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei
  
  ACM Multimedia 2022 | October 2022
  
  Publication Project
- DiT: Self-supervised Pre-training for Document Image Transformer
  
  Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei
  
  ACM Multimedia 2022 | October 2022
  
  Publication Project
- SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
  
  Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, Furu Wei
  
  EMNLP 2022 | October 2022
  
  Publication
- Separating Long-Form Speech with Group-Wise Permutation Invariant Training
  
  Wangyou Zhang, Zhuo Chen, Naoyuki Kanda, Shujie Liu, Jinyu Li, Sefik Emre Eskimez, Takuya Yoshioka, Xiong Xiao, Zhong Meng, Yanmin Qian, Furu Wei
  
  Interspeech 2022 | September 2022
  
  Publication
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
  
  Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, Furu Wei
  
  IEEE Journal of Selected Topics in Signal Processing | July 2022, Vol 16(6): pp. 1505-1518
  
  Publication
- Lossless Acceleration for Seq2seq Generation with Aggressive Decoding
  
  Tao Ge, Heming Xia, Xin Sun, Si-Qing Chen, Furu Wei
  
  May 2022
  
  Publication
- Neural Label Search for Zero-Shot Multi-Lingual Extractive Summarization
  
  Ruipeng Jia, Xingxing Zhang, Yanan Cao, Shi Wang, Zheng Lin, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding
  
  Junlong Li, Yiheng Xu, Lei Cui, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication Project
- BEiT: BERT Pre-Training of Image Transformers
  
  Hangbo Bao, Li Dong, Songhao Piao, Furu Wei
  
  ICLR 2022 | April 2022
  
  Publication
- Sequence Level Contrastive Learning for Text Summarization
  
  Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei
  
  AAAI 2022 | January 2022
  
  Publication
- Document AI: Benchmarks, Models and Applications
  
  Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
  
  November 2021
  
  Publication Project
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication Project
- LayoutReader: Pre-training of Text and Layout for Reading Order Detection
  
  Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, Furu Wei
  
  EMNLP 2021 | November 2021
  
  Publication Project
- Jointly Learning to Repair Code and Generate Commit Message
  
  Jiaqi Bai, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Zhoujun Li
  
  2021 Empirical Methods in Natural Language Processing | September 2021
  
  Publication
- Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
  
  Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei
  
  2021 Empirical Methods in Natural Language Processing | September 2021
  
  Publication
- Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
  
  Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian McAuley, Furu Wei
  
  2021 Empirical Methods in Natural Language Processing | September 2021
  
  Publication
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication Project
- Document AI: Benchmarks, Models and Applications (In Chinese)
  
  Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
  
  CCL 2021 | August 2021
  
  Project
- UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
  
  Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang
  
  2021 International Conference on Machine Learning | July 2021
  
  Publication
- Multilingual Agreement for Multilingual Neural Machine Translation
  
  Jian Yang, Yuwei Yin, Shuming Ma, Haoyang Huang, Dongdong Zhang, Zhoujun Li, Furu Wei
  
  July 2021
- Grammar-Based Patches Generation for Automated Program Repair
  
  Yu Tang, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Muyun Yang
  
  ACL-IJCNLP 2021 | July 2021
  
  PDF
- Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
  
  Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | July 2021
  
  Publication Github
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication Project
- Learning to Sample Replacements for ELECTRA Pre-Training
  
  Yaru Hao, Li Dong, Hangbo Bao, Ke Xu, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication Project
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication Project
- Consistency Regularization for Cross-Lingual Fine-Tuning
  
  Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
  
  Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
  
  Xin Sun, Tao Ge, Furu Wei, Houfeng Wang
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization
  
  Tengchao Lv, Lei Cui, Momcilo Vasilijevic, Furu Wei
  
  June 2021
  
  Preprint
- Memory-Efficient Differentiable Transformer Architecture Search
  
  Yuekai Zhao, Li Dong, Yelong Shen, Zhihua Zhang, Furu Wei, Wei Chen
  
  ACL-IJCNLP 2021 | May 2021
  
  Publication
- Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
  
  Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian J. McAuley, Furu Wei
  
  April 2021
  
  Publication
- LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
  
  Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
  
  April 2021
  
  Publication Project
- XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.
  
  Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko I. Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei
  
  December 2020
  
  Publication Publication Publication Project
- MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
  
  Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | December 2020
  
  PDF Publication
- DocBank: A Benchmark Dataset for Document Layout Analysis
  
  Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, Ming Zhou
  
  COLING 2020 | December 2020
  
  Publication Project
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
  
  Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao
  
  ECCV | August 2020
  
  Publication Video Github Project Project Project
- LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  
  Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
  
  KDD 2020 | August 2020
  
  PDF Publication Github Project
- UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
  
  Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
  
  37th International Conference on Machine Learning (ICML 2020) | July 2020
  
  Publication Github Project
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
  
  Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
  
  July 2020
  
  Publication Publication Github
- Multimodal Matching Transformer for Live Commenting
  
  Chaoqun Duan, Lei Cui, Shuming Ma, Furu Wei, Conghui Zhu, Tiejun Zhao
  
  ECAI 2020 | July 2020
  
  Publication
- TableBank: Table Benchmark for Image-based Table Detection and Recognition
  
  Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, Zhoujun Li
  
  LREC 2020 | May 2020
  
  Publication Project
- Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
  
  Yaru Hao, Li Dong, Furu Wei, Ke Xu
  
  AAAI 2021 | April 2020
  
  AAAI 2021 Best Paper Runner Up
  
  Publication PDF
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
  
  Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, Ming Zhou
  
  NeurIPS 2020 | February 2020
  
  Publication Project
- Unified Language Model Pre-training for Natural Language Understanding and Generation
  
  Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
  
  33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019
  
  Publication Github Project
- Neural Melody Composition from Lyrics
  
  Hangbo Bao, Shaohan Huang, Furu Wei, Lei Cui, Yu Wu, Chuanqi Tan, Songhao Piao, Ming Zhou
  
  NLPCC 2019 | September 2019
  
  Publication
- LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
  
  Shuming Ma, Lei Cui, Damai Dai, Furu Wei, Xu Sun
  
  AAAI 2019 | July 2019
  
  Publication Publication
- Retrieval-Enhanced Adversarial Training for Neural Response Generation
  
  Qingfu Zhu, Lei Cui, Wei-Nan Zhang, Furu Wei, Ting Liu
  
  ACL 2019 | July 2019
  
  Publication
- SeRI: A Dataset for Sub-event Relation Inference from an Encyclopedia
  
  Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
  
  NLPCC 2018 | August 2018
  
  Publication
- SuperAgent: A Customer Service Chatbot for E-commerce Websites
  
  Lei Cui, Furu Wei, Shaohan Huang, Chuanqi Tan, Chaoqun Duan, Ming Zhou
  
  ACL 2017 | July 2017
  
  Publication Publication
- Chain-of-Retrieval Augmented Generation
  
  Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, Furu Wei
  
  NeurIPS 2025 | October 2025
  
  Publication
- Scaling Laws of Synthetic Data for Language Models
  
  Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R. Fung, Weizhu Chen, Minhao Cheng, Furu Wei
  
  COLM 2025 | July 2025
  
  Publication
- Chain-of-Reasoning: Towards Unified Mathematical Reasoning in Large Language Models via a Multi-Paradigm Perspective
  
  Yiyao Yu, Yuxiang Zhang, Dongdong Zhang, Xiao Liang, Hengyuan Zhang, Xingxing Zhang, Mahmoud Khademi, Hany Awadalla, Junjie Wang, Yujiu Yang, Furu Wei
  
  2025 Meeting of the Association for Computational Linguistics | May 2025
  
  Publication Github
- Boosting Large Language Model for Speech Synthesis: An Empirical Study
  
  Hongkun Hao, Long Zhou, Shujie Liu, Jinyu Li, Shujie Hu, Rui Wang, Furu Wei
  
  ICASSP | April 2025
  
  Publication
- Examining False Positives under Inference Scaling for Mathematical Reasoning
  
  Yu Wang, Nan Yang, Liang Wang, Furu Wei
  
  MSR-TR-2025-2 | February 2025
  
  Published by Microsoft
  
  Publication
- Bootstrap Your Own Context Length
  
  Liang Wang, Nan Yang, Xingxing Zhang, Xiaolong Huang, Furu Wei
  
  ArXiv | December 2024, Vol abs/2412.18860
  
  DOI Publication Publication
- Little Giants: Synthesizing High-Quality Embedding Data at Scale
  
  Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, Zhicheng Dou
  
  NAACL 2025 | October 2024
  
  DOI Publication Publication
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
  
  Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
  
  June 2024
  
  Publication Project
- LongEmbed: Extending Embedding Models for Long Context Retrieval
  
  Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
  
  EMNLP 2024 | April 2024
  
  DOI Publication Publication
- Mixture of LoRA Experts
  
  Shaohan Huang, Furu Wei
  
  ICLR 2024 | April 2024
  
  Publication
- SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data
  
  Ziqiang Zhang, Sanyuan Chen, Long Zhou, Yu Wu, Shuo Ren, Shujie Liu, Zhuoyuan Yao, Xun Gong, Lirong Dai, Jinyu Li, Furu Wei
  
  IEEE/ACM Transactions on Audio, Speech, and Language Processing | March 2024
  
  Publication
- Generative Representational Instruction Tuning
  
  Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, Douwe Kiela
  
  ICLR 2025 | February 2024
  
  DOI Publication Publication
- Improving Text Embeddings with Large Language Models
  
  Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
  
  ACL 2024 | January 2024
  
  Preprint
- Large Search Model: Redefining Search Stack in the Era of LLMs
  
  Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
  
  SIGIR Forum | December 2023, Vol 57(2)
  
  Publication
- Fine-Tuning LLaMA for Multi-Stage Text Retrieval
  
  Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, Jimmy Lin
  
  SIGIR 2024 | October 2023
  
  PDF
- PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
  
  Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
  
  ICLR 2024 | October 2023
  
  Publication
- Kosmos-2.5: A Multimodal Literate Model
  
  Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
  
  September 2023
  
  Preprint Project
- Generative Retrieval for Conversational Question Answering
  
  Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
  
  Information Processing and Management | August 2023, Vol 60(5)
  
  Publication
- Learning to Retrieve In-Context Examples for Large Language Models
  
  Liang Wang, Nan Yang, Furu Wei
  
  EACL 2024 | July 2023
  
  Publication
- Learning to Rank in Generative Retrieval
  
  Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
  
  AAAI 2024 | June 2023
  
  Publication
- VatLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
  
  Qiushi Zhu, Long Zhou, Ziqiang Zhang, Shujie Liu, Binxing Jiao, Jie Zhang, Lirong Dai, Daxin Jiang, Jinyu Li, Furu Wei
  
  IEEE Transactions on Multimedia | May 2023
  
  Publication
- TextDiffuser: Diffusion Models as Text Painters
  
  Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei
  
  NeurIPS 2023 | May 2023
  
  Publication
- Multiview Identifiers Enhanced Generative Retrieval
  
  Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
  
  ACL 2023 | May 2023
  
  Publication
- Inference with Reference: Lossless Acceleration of Large Language Models
  
  Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang (姜大昕), Linjun Yang, Rangan Majumder, Furu Wei
  
  MSR-TR-2023-17 | April 2023
  
  Published by Microsoft
  
  Publication
- Query2doc: Query Expansion with Large Language Models
  
  Liang Wang, Nan Yang, Furu Wei
  
  EMNLP 2023 | March 2023
  
  Publication
- TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
  
  Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei
  
  AAAI 2023 | February 2023
  
  Publication Project
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
  
  Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang (姜大昕), Rangan Majumder, Furu Wei
  
  December 2022
  
  Publication
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
  
  Yupang Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei
  
  ACM Multimedia 2022 | October 2022
  
  Publication Project
- SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
  
  Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, Furu Wei
  
  EMNLP 2022 | October 2022
  
  Publication
- Separating Long-Form Speech with Group-Wise Permutation Invariant Training
  
  Wangyou Zhang, Zhuo Chen, Naoyuki Kanda, Shujie Liu, Jinyu Li, Sefik Emre Eskimez, Takuya Yoshioka, Xiong Xiao, Zhong Meng, Yanmin Qian, Furu Wei
  
  Interspeech 2022 | September 2022
  
  Publication
- Learning Diverse Document Representations with Deep Query Interactions for Dense Retrieval
  
  Zehan Li, Nan Yang, Liang Wang, Furu Wei
  
  August 2022
  
  Publication
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
  
  Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, Furu Wei
  
  IEEE Journal of Selected Topics in Signal Processing | July 2022, Vol 16(6): pp. 1505-1518
  
  Publication
- SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
  
  Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang (姜大昕), Rangan Majumder, Furu Wei
  
  ACL 2023 | July 2022
  
  Publication
- XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding
  
  Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication Project
- Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
  
  Guanhua Chen, Shuming Ma, Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
  
  Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang (姜大昕), Haoyi Zhou, Jianxin Li, Furu Wei
  
  May 2022
- StableMoE: Stable Routing Strategy for Mixture of Experts
  
  Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
  
  Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- Neural Label Search for Zero-Shot Multi-Lingual Extractive Summarization
  
  Ruipeng Jia, Xingxing Zhang, Yanan Cao, Shi Wang, Zheng Lin, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- Knowledge Neurons in Pretrained Transformers
  
  Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- Controllable Natural Language Generation with Contrastive Prefixes
  
  Jing Qian, Li Dong, Yelong Shen, Furu Wei, Weizhu Chen
  
  ACL 2022 | May 2022
  
  Publication
- CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment
  
  Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- Attention Temperature Matters in Abstractive Summarization Distillation
  
  Shengqiang Zhang, Xingxing Zhang, Hangbo Bao, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding
  
  Junlong Li, Yiheng Xu, Lei Cui, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication Project
- Document AI: Benchmarks, Models and Applications
  
  Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
  
  November 2021
  
  Publication Project
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication Project
- LayoutReader: Pre-training of Text and Layout for Reading Order Detection
  
  Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, Furu Wei
  
  EMNLP 2021 | November 2021
  
  Publication Project
- MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
  
  Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  EMNLP 2021 | October 2021
  
  Publication
- Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
  
  Guanhua Chen, Shuming Ma, Yun Chen, Li Dong, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei
  
  EMNLP 2021 | October 2021
  
  Publication
- Jointly Learning to Repair Code and Generate Commit Message
  
  Jiaqi Bai, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Zhoujun Li
  
  2021 Empirical Methods in Natural Language Processing | September 2021
  
  Publication
- Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
  
  Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei
  
  2021 Empirical Methods in Natural Language Processing | September 2021
  
  Publication
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication Project
- Document AI: Benchmarks, Models and Applications (In Chinese)
  
  Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
  
  CCL 2021 | August 2021
  
  Project
- UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
  
  Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang
  
  2021 International Conference on Machine Learning | July 2021
  
  Publication
- Grammar-Based Patches Generation for Automated Program Repair
  
  Yu Tang, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Muyun Yang
  
  ACL-IJCNLP 2021 | July 2021
  
  PDF
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication Project
- Learning to Sample Replacements for ELECTRA Pre-Training
  
  Yaru Hao, Li Dong, Hangbo Bao, Ke Xu, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication Project
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication Project
- Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
  
  Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization
  
  Tengchao Lv, Lei Cui, Momcilo Vasilijevic, Furu Wei
  
  June 2021
  
  Preprint
- LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
  
  Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
  
  April 2021
  
  Publication Project
- LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
  
  Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou
  
  ACL-IJCNLP 2021 | January 2021
  
  Preprint Publication Github Project
- XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.
  
  Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko I. Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei
  
  December 2020
  
  Publication Publication Publication Project
- DocBank: A Benchmark Dataset for Document Layout Analysis
  
  Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, Ming Zhou
  
  COLING 2020 | December 2020
  
  Publication Project
- LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  
  Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
  
  KDD 2020 | August 2020
  
  PDF Publication Github Project
- Multimodal Matching Transformer for Live Commenting
  
  Chaoqun Duan, Lei Cui, Shuming Ma, Furu Wei, Conghui Zhu, Tiejun Zhao
  
  ECAI 2020 | July 2020
  
  Publication
- TableBank: Table Benchmark for Image-based Table Detection and Recognition
  
  Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, Zhoujun Li
  
  LREC 2020 | May 2020
  
  Publication Project
- Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
  
  Yaru Hao, Li Dong, Furu Wei, Ke Xu
  
  AAAI 2021 | April 2020
  
  AAAI 2021 Best Paper Runner Up
  
  Publication PDF
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
  
  Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai
  
  Eighth International Conference on Learning Representations (ICLR) | April 2020
  
  Publication PDF Github
- Self-Adversarial Learning with Comparative Discrimination for Text Generation
  
  Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, Ming Zhou
  
  Eighth International Conference on Learning Representations (ICLR) | April 2020
  
  Publication PDF
- Unified Language Model Pre-training for Natural Language Understanding and Generation
  
  Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
  
  33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019
  
  Publication Github Project
- Neural Melody Composition from Lyrics
  
  Hangbo Bao, Shaohan Huang, Furu Wei, Lei Cui, Yu Wu, Chuanqi Tan, Songhao Piao, Ming Zhou
  
  NLPCC 2019 | September 2019
  
  Publication
- LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
  
  Shuming Ma, Lei Cui, Damai Dai, Furu Wei, Xu Sun
  
  AAAI 2019 | July 2019
  
  Publication Publication
- Retrieval-Enhanced Adversarial Training for Neural Response Generation
  
  Qingfu Zhu, Lei Cui, Wei-Nan Zhang, Furu Wei, Ting Liu
  
  ACL 2019 | July 2019
  
  Publication
- Fine-grained Coordinated Cross-lingual Text Stream Alignment for Endless Language Knowledge Acquisition
  
  Tao Ge, Qing Dou, Heng Ji, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
  
  EMNLP 2018 | November 2018
- SeRI: A Dataset for Sub-event Relation Inference from an Encyclopedia
  
  Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
  
  NLPCC 2018 | August 2018
  
  Publication
- Neural Open Information Extraction
  
  Lei Cui, Furu Wei, Ming Zhou
  
  ACL 2018 | July 2018
  
  Publication Publication
- Attention-Fused Deep Matching Network for Natural Language Inference
  
  Chaoqun Duan, Lei Cui, Xinchi Chen, Furu Wei, Conghui Zhu, Tiejun Zhao
  
  IJCAI 2018 | July 2018
  
  Publication
- EventWiki: A Knowledge Base of Major Events
  
  Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
  
  LREC 2018 | May 2018
  
  Publication Publication Publication
- SuperAgent: A Customer Service Chatbot for E-commerce Websites
  
  Lei Cui, Furu Wei, Shaohan Huang, Chuanqi Tan, Chaoqun Duan, Ming Zhou
  
  ACL 2017 | July 2017
  
  Publication Publication
- Chain-of-Retrieval Augmented Generation
  
  Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, Furu Wei
  
  NeurIPS 2025 | October 2025
  
  Publication
- Little Giants: Synthesizing High-Quality Embedding Data at Scale
  
  Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, Zhicheng Dou
  
  NAACL 2025 | October 2024
  
  DOI Publication Publication
- LongEmbed: Extending Embedding Models for Long Context Retrieval
  
  Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
  
  EMNLP 2024 | April 2024
  
  DOI Publication Publication
- Generative Representational Instruction Tuning
  
  Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, Douwe Kiela
  
  ICLR 2025 | February 2024
  
  DOI Publication Publication
- Improving Text Embeddings with Large Language Models
  
  Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
  
  ACL 2024 | January 2024
  
  Preprint
- Large Search Model: Redefining Search Stack in the Era of LLMs
  
  Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
  
  SIGIR Forum | December 2023, Vol 57(2)
  
  Publication
- Fine-Tuning LLaMA for Multi-Stage Text Retrieval
  
  Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, Jimmy Lin
  
  SIGIR 2024 | October 2023
  
  PDF
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
  
  Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang (姜大昕), Rangan Majumder, Furu Wei
  
  December 2022
  
  Publication
- Leveraging Contextual Sentence Relations for Extractive Summarization Using a Neural Attention Model
  
  Pengjie Ren, Zhumin Chen, Zhaochun Ren, Furu Wei, Jun Ma, Maarten de Rijke
  
  SIGIR '17, Shinjuku, Tokyo, Japan, August 07-11, 2017 | August 2017
  
  Publication
- SuperAgent: A Customer Service Chatbot for E-commerce Websites
  
  Lei Cui, Furu Wei, Shaohan Huang, Chuanqi Tan, Chaoqun Duan, Ming Zhou
  
  ACL 2017 | July 2017
  
  Publication Publication
- QuickView: advanced search of tweets
  
  Xiaohua Liu, Long Jiang, Furu Wei, Ming Zhou, QuickView Team Microsoft
  
  SIGIR (Demo paper) | January 2011
  
  ISBN: 978-1-4503-0757-4
  
  Publication Publication
- REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models
  
  Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang
  
  December 2024
  
  Publication
- OpinionSeer: Interactive Visualization of Hotel Customer Feedback
  
  Yingcai Wu, Furu Wei, Shixia Liu, Norman Au, Weiwei Cui, Hong Zhou, Huamin Qu
  
  IEEE Transactions on Visualization and Computer Graphics | November 2010, pp. 1109-1118
  
  Publication Publication
- Context-Preserving, Dynamic Word Cloud Visualization
  
  Weiwei Cui, Yingcai Wu, Shixia Liu, Furu Wei, Michelle Zhou, Huamin Qu
  
  IEEE Computer Graphics and Applications | September 2010, Vol 30(6): pp. 42
  
  Publication Publication
- Context Preserving Dynamic Word Cloud Visualization
  
  Weiwei Cui, Yingcai Wu, Shixia Liu, Furu Wei, Michelle Zhou, Huamin Qu
  
  2010 IEEE Pacific Visualization Symposium (PacificVis) | March 2010
  
  Publication Publication
- PEACE: Empowering Geologic Map Holistic Understanding with MLLMs
  
  Yangyu Huang, Tianyi Gao, Haoran Xu, Qihao Zhao, Yang Song, Zhipeng Gui, Tengchao Lv, Hao Cheng, Lei Cui, Scarlett Li, Furu Wei
  
  January 2025
  
  Publication Access
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Kosmos-2.5: A Multimodal Literate Model
  
  Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
  
  September 2023
  
  Preprint Project
- TextDiffuser: Diffusion Models as Text Painters
  
  Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei
  
  NeurIPS 2023 | May 2023
  
  Publication
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
  
  Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei
  
  AAAI 2023 | February 2023
  
  Publication Project
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
  
  Yupang Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei
  
  ACM Multimedia 2022 | October 2022
  
  Publication Project
- DiT: Self-supervised Pre-training for Document Image Transformer
  
  Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei
  
  ACM Multimedia 2022 | October 2022
  
  Publication Project
- Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
  
  Wenhui Wang, Hangbo Bao, Li Dong, Kriti Aggarwal, Saksham Singhal, Subhojit Som, Furu Wei, Johan Bjorck, Zhiliang Peng, Qiang Liu, Owais Khan Mohammed
  
  August 2022
  
  Publication
- Document AI: Benchmarks, Models and Applications
  
  Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
  
  November 2021
  
  Publication Project
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
  
  Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao
  
  ECCV | August 2020
  
  Publication Video Github Project Project Project
- TableBank: Table Benchmark for Image-based Table Detection and Recognition
  
  Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, Zhoujun Li
  
  LREC 2020 | May 2020
  
  Publication Project
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
  
  Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai
  
  Eighth International Conference on Learning Representations (ICLR) | April 2020
  
  Publication PDF Github
- PEACE: Empowering Geologic Map Holistic Understanding with MLLMs
  
  Yangyu Huang, Tianyi Gao, Haoran Xu, Qihao Zhao, Yang Song, Zhipeng Gui, Tengchao Lv, Hao Cheng, Lei Cui, Scarlett Li, Furu Wei
  
  January 2025
  
  Publication Access
- A whole-slide foundation model for digital pathology from real-world data
  
  Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, Yanbo Xu, Mu-Hsin Wei, Wenhui Wang, Shuming Ma, Furu Wei, Jianwei Yang, Chunyuan Li, Jianfeng Gao, Jaylen Rosemon, Tucker Bower, Soohee Lee, R. Weerasinghe, Bill Wright, Ari Robicsek, Brian Piening, Carlo Bifulco, Sheng Wang, Hoifung Poon
  
  Nature | May 2024
  
  Publication DOI Publication
- When an Image is Worth 1,024 x 1,024 Words: A Case Study in Computational Pathology
  
  Wenhui Wang, Shuming Ma, Hanwen Xu, Naoto Usuyama, Jiayu Ding, Hoifung Poon, Furu Wei
  
  December 2023
  
  DOI Publication Publication
- Autoregressive Speech Synthesis without Vector Quantization
  
  Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, Furu Wei
  
  July 2024
  
  Publication Project
- VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
  
  Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, Furu Wei
  
  June 2024
  
  Publication Project
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
  
  Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
  
  June 2024
  
  DOI Publication Publication Project
- WavLLM: Towards Robust and Adaptive Speech Large Language Model
  
  Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei
  
  March 2024
  
  DOI Publication Publication
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
  
  Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, Furu Wei
  
  January 2023
  
  Publication PDF Project
- SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
  
  Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, Furu Wei
  
  EMNLP 2022 | October 2022
  
  Publication
- Separating Long-Form Speech with Group-Wise Permutation Invariant Training
  
  Wangyou Zhang, Zhuo Chen, Naoyuki Kanda, Shujie Liu, Jinyu Li, Sefik Emre Eskimez, Takuya Yoshioka, Xiong Xiao, Zhong Meng, Yanmin Qian, Furu Wei
  
  Interspeech 2022 | September 2022
  
  Publication
- UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
  
  Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, Xiangzhan Yu
  
  ICASSP 2022 | May 2022
  
  Publication
- MathScale: Scaling Instruction Tuning for Mathematical Reasoning
  
  Zhengyang Tang, Xingxing Zhang, Benyou Wang, Furu Wei
  
  ICML 2024 | March 2024
  
  DOI Publication Publication
- Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
  
  Xin Sun, Tao Ge, Furu Wei, Houfeng Wang
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
  
  Yaru Hao, Li Dong, Furu Wei, Ke Xu
  
  AAAI 2021 | April 2020
  
  AAAI 2021 Best Paper Runner Up
  
  Publication PDF

- Bootstrap Your Own Context Length
  
  Liang Wang, Nan Yang, Xingxing Zhang, Xiaolong Huang, Furu Wei
  
  ArXiv | December 2024, Vol abs/2412.18860
  
  DOI Publication Publication
- A whole-slide foundation model for digital pathology from real-world data
  
  Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, Yanbo Xu, Mu-Hsin Wei, Wenhui Wang, Shuming Ma, Furu Wei, Jianwei Yang, Chunyuan Li, Jianfeng Gao, Jaylen Rosemon, Tucker Bower, Soohee Lee, R. Weerasinghe, Bill Wright, Ari Robicsek, Brian Piening, Carlo Bifulco, Sheng Wang, Hoifung Poon
  
  Nature | May 2024
  
  Publication DOI Publication
- SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data
  
  Ziqiang Zhang, Sanyuan Chen, Long Zhou, Yu Wu, Shuo Ren, Shujie Liu, Zhuoyuan Yao, Xun Gong, Lirong Dai, Jinyu Li, Furu Wei
  
  IEEE/ACM Transactions on Audio, Speech, and Language Processing | March 2024
  
  Publication
- Large Search Model: Redefining Search Stack in the Era of LLMs
  
  Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
  
  SIGIR Forum | December 2023, Vol 57(2)
  
  Publication
- Generative Retrieval for Conversational Question Answering
  
  Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
  
  Information Processing and Management | August 2023, Vol 60(5)
  
  Publication
- Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting
  
  Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, Furu Wei
  
  2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) | May 2023, pp. 12365-12394
  
  DOI Publication Publication Publication
- VatLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
  
  Qiushi Zhu, Long Zhou, Ziqiang Zhang, Shujie Liu, Binxing Jiao, Jie Zhang, Lirong Dai, Daxin Jiang, Jinyu Li, Furu Wei
  
  IEEE Transactions on Multimedia | May 2023
  
  Publication
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
  
  Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, Furu Wei
  
  IEEE Journal of Selected Topics in Signal Processing | July 2022, Vol 16(6): pp. 1505-1518
  
  Publication
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication Project
- OpinionSeer: Interactive Visualization of Hotel Customer Feedback
  
  Yingcai Wu, Furu Wei, Shixia Liu, Norman Au, Weiwei Cui, Hong Zhou, Huamin Qu
  
  IEEE Transactions on Visualization and Computer Graphics | November 2010, pp. 1109-1118
  
  Publication Publication
- Context-Preserving, Dynamic Word Cloud Visualization
  
  Weiwei Cui, Yingcai Wu, Shixia Liu, Furu Wei, Michelle Zhou, Huamin Qu
  
  IEEE Computer Graphics and Applications | September 2010, Vol 30(6): pp. 42
  
  Publication Publication
- DocReward: A Document Reward Model for Structuring and Stylizing
  
  Junpeng Liu, Yuzhong Zhao, Bowen Cao, Jiayu Ding, Yilin Jia, Tengchao Lv, Yupan Huang, Shaohan Huang, Nan Yang, Li Dong, Lei Cui, Tao Ge, Xun Wang, Huitian Jiao, Sun Mao, Fnu Kartik, Siqing Chen, Wai Lam, Furu Wei
  
  October 2025
  
  Publication
- MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark
  
  Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Furu Wei, Qinzheng Sun, Ying Xin, Shaoguang Mao, Xin Zhang, Qiufeng Yin, Scarlett Li
  
  December 2024
  
  Publication Github
- BitNet a4.8: 4-bit Activations for 1-bit LLMs
  
  Hongyu Wang, Shuming Ma, Furu Wei
  
  November 2024
  
  Work in progress
  
  Publication Access
- 1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs
  
  Jinheng Wang, Hansong Zhou, Ting Song, Shaoguang Mao, Shuming Ma, Hongyu Wang, Yan Xia, Furu Wei
  
  October 2024
  
  Publication
- Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
  
  Hongyu Wang, Shuming Ma, Ruiping Wang, Furu Wei
  
  July 2024
  
  Publication
- Autoregressive Speech Synthesis without Vector Quantization
  
  Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, Furu Wei
  
  July 2024
  
  Publication Project
- VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
  
  Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, Furu Wei
  
  June 2024
  
  Publication Project
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
  
  Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
  
  June 2024
  
  Publication Project
- Direct Preference Knowledge Distillation for Large Language Models
  
  Yixing Li, Yuxian Gu, Li Dong, Dequan Wang, Yu Cheng, Furu Wei
  
  June 2024
  
  DOI Publication Publication
- Instruction Pre-Training: Language Models are Supervised Multitask Learners
  
  Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei
  
  June 2024
  
  DOI Publication Publication
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
  
  Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
  
  June 2024
  
  DOI Publication Publication Project
- MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
  
  Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
  
  May 2024
  
  DOI Publication Publication
- LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models
  
  Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, Furu Wei
  
  March 2024
  
  DOI Publication Publication
- WavLLM: Towards Robust and Adaptive Speech Large Language Model
  
  Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei
  
  March 2024
  
  DOI Publication Publication
- ResLoRA: Identity Residual Mapping in Low-Rank Adaption
  
  Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  February 2024
  
  DOI Publication Publication
- Towards Optimal Learning of Language Models
  
  Yuxian Gu, Li Dong, Yaru Hao, Qingxiu Dong, Minlie Huang, Furu Wei
  
  February 2024
  
  DOI Publication Publication
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
  
  Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
  
  Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- When an Image is Worth 1,024 x 1,024 Words: A Case Study in Computational Pathology
  
  Wenhui Wang, Shuming Ma, Hanwen Xu, Naoto Usuyama, Jiayu Ding, Hoifung Poon, Furu Wei
  
  December 2023
  
  DOI Publication Publication
- BitNet: Scaling 1-bit Transformers for Large Language Models
  
  Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei
  
  October 2023
  
  DOI Publication
- Retentive Network: A Successor to Transformer for Large Language Models
  
  Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei
  
  August 2023
  
  Publication
- LongNet: Scaling Transformers to 1,000,000,000 Tokens
  
  Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei
  
  July 2023
  
  DOI PDF Video
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
  
  Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, Furu Wei
  
  January 2023
  
  Publication PDF Project
- Learning Diverse Document Representations with Deep Query Interactions for Dense Retrieval
  
  Zehan Li, Nan Yang, Liang Wang, Furu Wei
  
  August 2022
  
  Publication
- Lossless Acceleration for Seq2seq Generation with Aggressive Decoding
  
  Tao Ge, Heming Xia, Xin Sun, Si-Qing Chen, Furu Wei
  
  May 2022
  
  Publication
- Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
  
  Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian J. McAuley, Furu Wei
  
  April 2021
  
  Publication
- XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.
  
  Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko I. Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei
  
  December 2020
  
  Publication Publication Publication Project
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
  
  Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
  
  July 2020
  
  Publication Publication Github
- Chain-of-Retrieval Augmented Generation
  
  Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, Furu Wei
  
  NeurIPS 2025 | October 2025
  
  Publication
- Native Hybrid Thinking Models
  
  Lingjie Jiang, Xun Wu, Shaohan Huang, Qingxiu Dong, Zewen Chi, Li Dong, Xingxing Zhang, Tengchao Lv, Lei Cui, Furu Wei
  
  NeurIPS 2025 | October 2025
  
  Publication
- Scaling Laws of Synthetic Data for Language Models
  
  Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R. Fung, Weizhu Chen, Minhao Cheng, Furu Wei
  
  COLM 2025 | July 2025
  
  Publication
- Reward Reasoning Model
  
  Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, Furu Wei
  
  NeurIPS 2025 | May 2025
  
  DOI Publication Publication
- Chain-of-Reasoning: Towards Unified Mathematical Reasoning in Large Language Models via a Multi-Paradigm Perspective
  
  Yiyao Yu, Yuxiang Zhang, Dongdong Zhang, Xiao Liang, Hengyuan Zhang, Xingxing Zhang, Mahmoud Khademi, Hany Awadalla, Junjie Wang, Yujiu Yang, Furu Wei
  
  2025 Meeting of the Association for Computational Linguistics | May 2025
  
  Publication Github
- Boosting Large Language Model for Speech Synthesis: An Empirical Study
  
  Hongkun Hao, Long Zhou, Shujie Liu, Jinyu Li, Shujie Hu, Rui Wang, Furu Wei
  
  ICASSP | April 2025
  
  Publication
- Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning
  
  Wenkai Yang, Shuming Ma, Yankai Lin, Furu Wei
  
  NeurIPS 2025 | February 2025
  
  DOI Publication Publication
- PEACE: Empowering Geologic Map Holistic Understanding with MLLMs
  
  Yangyu Huang, Tianyi Gao, Haoran Xu, Qihao Zhao, Yang Song, Zhipeng Gui, Tengchao Lv, Hao Cheng, Lei Cui, Scarlett Li, Furu Wei
  
  January 2025
  
  Publication Access
- Imagine while Reasoning in Space: Multimodal Visualization-of-Thought
  
  Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vuli'c, Furu Wei
  
  ICML 2025 | January 2025
  
  DOI Publication Publication
- Little Giants: Synthesizing High-Quality Embedding Data at Scale
  
  Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, Zhicheng Dou
  
  NAACL 2025 | October 2024
  
  DOI Publication Publication
- xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
  
  Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, Dongyan Zhao
  
  NeurIPS 2024 | May 2024
  
  DOI Publication Publication
- You Only Cache Once: Decoder-Decoder Architectures for Language Models
  
  Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei
  
  NeurIPS 2024 | May 2024
  
  DOI Publication Publication
- Kosmos-G: Generating Images in Context with Multimodal Large Language Models
  
  Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei
  
  ICLR 2024 | May 2024
  
  DOI Publication PDF
- Multi-Head Mixture-of-Experts
  
  Xun Wu, Shaohan Huang, Wenhui Wang, Furu Wei
  
  NeurIPS 2024 | April 2024
  
  DOI Publication Publication
- LongEmbed: Extending Embedding Models for Long Context Retrieval
  
  Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
  
  EMNLP 2024 | April 2024
  
  DOI Publication Publication
- Mind’s Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models
  
  Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, Furu Wei
  
  NeurIPS 2024 | April 2024
  
  Publication
- Mixture of LoRA Experts
  
  Shaohan Huang, Furu Wei
  
  ICLR 2024 | April 2024
  
  Publication
- MathScale: Scaling Instruction Tuning for Mathematical Reasoning
  
  Zhengyang Tang, Xingxing Zhang, Benyou Wang, Furu Wei
  
  ICML 2024 | March 2024
  
  DOI Publication Publication
- Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models
  
  Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, Ji-Rong Wen
  
  ACL 2024 | February 2024
  
  DOI Publication Publication
- Generative Representational Instruction Tuning
  
  Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, Douwe Kiela
  
  ICLR 2025 | February 2024
  
  DOI Publication Publication
- Text Diffusion with Reinforced Conditioning
  
  Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  AAAI 2024 | February 2024
  
  Publication
- Improving Text Embeddings with Large Language Models
  
  Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
  
  ACL 2024 | January 2024
  
  Preprint
- Fine-Tuning LLaMA for Multi-Stage Text Retrieval
  
  Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, Jimmy Lin
  
  SIGIR 2024 | October 2023
  
  PDF
- Augmenting Language Models with Long-Term Memory
  
  Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, Furu Wei
  
  NeurIPS 2023 | October 2023
  
  Publication
- PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
  
  Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
  
  ICLR 2024 | October 2023
  
  Publication
- Adapting Large Language Models via Reading Comprehension
  
  Daixuan Cheng, Shaohan Huang, Furu Wei
  
  ICLR 2024 | September 2023
  
  DOI
- In-context Autoencoder for Context Compression in a Large Language Model
  
  Tao Ge, Jing Hu, Haixun Wang, Si-Qing Chen, Furu Wei
  
  ICLR 2024 | July 2023
  
  DOI Publication
- Learning to Retrieve In-Context Examples for Large Language Models
  
  Liang Wang, Nan Yang, Furu Wei
  
  EACL 2024 | July 2023
  
  Publication
- Knowledge Distillation of Large Language Models
  
  Yuxian Gu, Li Dong, Furu Wei, Minlie Huang
  
  ICLR 2024 | June 2023
  
  DOI PDF
- Learning to Rank in Generative Retrieval
  
  Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
  
  AAAI 2024 | June 2023
  
  Publication
- TextDiffuser: Diffusion Models as Text Painters
  
  Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei
  
  NeurIPS 2023 | May 2023
  
  Publication
- Multiview Identifiers Enhanced Generative Retrieval
  
  Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
  
  ACL 2023 | May 2023
  
  Publication
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- Query2doc: Query Expansion with Large Language Models
  
  Liang Wang, Nan Yang, Furu Wei
  
  EMNLP 2023 | March 2023
  
  Publication
- TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
  
  Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei
  
  AAAI 2023 | February 2023
  
  Publication Project
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
  
  Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang (姜大昕), Rangan Majumder, Furu Wei
  
  December 2022
  
  Publication
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
  
  Yupang Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei
  
  ACM Multimedia 2022 | October 2022
  
  Publication Project
- DiT: Self-supervised Pre-training for Document Image Transformer
  
  Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei
  
  ACM Multimedia 2022 | October 2022
  
  Publication Project
- SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
  
  Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, Furu Wei
  
  EMNLP 2022 | October 2022
  
  Publication
- Separating Long-Form Speech with Group-Wise Permutation Invariant Training
  
  Wangyou Zhang, Zhuo Chen, Naoyuki Kanda, Shujie Liu, Jinyu Li, Sefik Emre Eskimez, Takuya Yoshioka, Xiong Xiao, Zhong Meng, Yanmin Qian, Furu Wei
  
  Interspeech 2022 | September 2022
  
  Publication
- SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
  
  Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang (姜大昕), Rangan Majumder, Furu Wei
  
  ACL 2023 | July 2022
  
  Publication
- UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
  
  Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, Xiangzhan Yu
  
  ICASSP 2022 | May 2022
  
  Publication
- XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding
  
  Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication Project
- Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
  
  Guanhua Chen, Shuming Ma, Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- StableMoE: Stable Routing Strategy for Mixture of Experts
  
  Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
  
  Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- Neural Label Search for Zero-Shot Multi-Lingual Extractive Summarization
  
  Ruipeng Jia, Xingxing Zhang, Yanan Cao, Shi Wang, Zheng Lin, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- Knowledge Neurons in Pretrained Transformers
  
  Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- Controllable Natural Language Generation with Contrastive Prefixes
  
  Jing Qian, Li Dong, Yelong Shen, Furu Wei, Weizhu Chen
  
  ACL 2022 | May 2022
  
  Publication
- CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment
  
  Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- Attention Temperature Matters in Abstractive Summarization Distillation
  
  Shengqiang Zhang, Xingxing Zhang, Hangbo Bao, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication
- MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding
  
  Junlong Li, Yiheng Xu, Lei Cui, Furu Wei
  
  ACL 2022 | May 2022
  
  Publication Project
- BEiT: BERT Pre-Training of Image Transformers
  
  Hangbo Bao, Li Dong, Songhao Piao, Furu Wei
  
  ICLR 2022 | April 2022
  
  Publication
- Sequence Level Contrastive Learning for Text Summarization
  
  Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei
  
  AAAI 2022 | January 2022
  
  Publication
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication Project
- LayoutReader: Pre-training of Text and Layout for Reading Order Detection
  
  Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, Furu Wei
  
  EMNLP 2021 | November 2021
  
  Publication Project
- MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
  
  Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  EMNLP 2021 | October 2021
  
  Publication
- Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
  
  Guanhua Chen, Shuming Ma, Yun Chen, Li Dong, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei
  
  EMNLP 2021 | October 2021
  
  Publication
- Jointly Learning to Repair Code and Generate Commit Message
  
  Jiaqi Bai, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Zhoujun Li
  
  2021 Empirical Methods in Natural Language Processing | September 2021
  
  Publication
- Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
  
  Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei
  
  2021 Empirical Methods in Natural Language Processing | September 2021
  
  Publication
- Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
  
  Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian McAuley, Furu Wei
  
  2021 Empirical Methods in Natural Language Processing | September 2021
  
  Publication
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication Project
- Document AI: Benchmarks, Models and Applications (In Chinese)
  
  Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
  
  CCL 2021 | August 2021
  
  Project
- UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
  
  Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang
  
  2021 International Conference on Machine Learning | July 2021
  
  Publication
- Grammar-Based Patches Generation for Automated Program Repair
  
  Yu Tang, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Muyun Yang
  
  ACL-IJCNLP 2021 | July 2021
  
  PDF
- Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
  
  Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | July 2021
  
  Publication Github
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication Project
- Learning to Sample Replacements for ELECTRA Pre-Training
  
  Yaru Hao, Li Dong, Hangbo Bao, Ke Xu, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- Consistency Regularization for Cross-Lingual Fine-Tuning
  
  Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
  
  Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
  
  Xin Sun, Tao Ge, Furu Wei, Houfeng Wang
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- Memory-Efficient Differentiable Transformer Architecture Search
  
  Yuekai Zhao, Li Dong, Yelong Shen, Zhihua Zhang, Furu Wei, Wei Chen
  
  ACL-IJCNLP 2021 | May 2021
  
  Publication
- LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
  
  Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou
  
  ACL-IJCNLP 2021 | January 2021
  
  Preprint Publication Github Project
- MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
  
  Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | December 2020
  
  PDF Publication
- DocBank: A Benchmark Dataset for Document Layout Analysis
  
  Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, Ming Zhou
  
  COLING 2020 | December 2020
  
  Publication Project
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
  
  Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao
  
  ECCV | August 2020
  
  Publication Video Github Project Project Project
- LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  
  Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
  
  KDD 2020 | August 2020
  
  PDF Publication Github Project
- UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
  
  Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
  
  37th International Conference on Machine Learning (ICML 2020) | July 2020
  
  Publication Github Project
- Multimodal Matching Transformer for Live Commenting
  
  Chaoqun Duan, Lei Cui, Shuming Ma, Furu Wei, Conghui Zhu, Tiejun Zhao
  
  ECAI 2020 | July 2020
  
  Publication
- TableBank: Table Benchmark for Image-based Table Detection and Recognition
  
  Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, Zhoujun Li
  
  LREC 2020 | May 2020
  
  Publication Project
- Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
  
  Yaru Hao, Li Dong, Furu Wei, Ke Xu
  
  AAAI 2021 | April 2020
  
  AAAI 2021 Best Paper Runner Up
  
  Publication PDF
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
  
  Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai
  
  Eighth International Conference on Learning Representations (ICLR) | April 2020
  
  Publication PDF Github
- Self-Adversarial Learning with Comparative Discrimination for Text Generation
  
  Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, Ming Zhou
  
  Eighth International Conference on Learning Representations (ICLR) | April 2020
  
  Publication PDF
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
  
  Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, Ming Zhou
  
  NeurIPS 2020 | February 2020
  
  Publication Project
- Unified Language Model Pre-training for Natural Language Understanding and Generation
  
  Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
  
  33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019
  
  Publication Github Project
- Neural Melody Composition from Lyrics
  
  Hangbo Bao, Shaohan Huang, Furu Wei, Lei Cui, Yu Wu, Chuanqi Tan, Songhao Piao, Ming Zhou
  
  NLPCC 2019 | September 2019
  
  Publication
- LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
  
  Shuming Ma, Lei Cui, Damai Dai, Furu Wei, Xu Sun
  
  AAAI 2019 | July 2019
  
  Publication Publication
- Retrieval-Enhanced Adversarial Training for Neural Response Generation
  
  Qingfu Zhu, Lei Cui, Wei-Nan Zhang, Furu Wei, Ting Liu
  
  ACL 2019 | July 2019
  
  Publication
- Fine-grained Coordinated Cross-lingual Text Stream Alignment for Endless Language Knowledge Acquisition
  
  Tao Ge, Qing Dou, Heng Ji, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
  
  EMNLP 2018 | November 2018
- SeRI: A Dataset for Sub-event Relation Inference from an Encyclopedia
  
  Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
  
  NLPCC 2018 | August 2018
  
  Publication
- Neural Open Information Extraction
  
  Lei Cui, Furu Wei, Ming Zhou
  
  ACL 2018 | July 2018
  
  Publication Publication
- Attention-Fused Deep Matching Network for Natural Language Inference
  
  Chaoqun Duan, Lei Cui, Xinchi Chen, Furu Wei, Conghui Zhu, Tiejun Zhao
  
  IJCAI 2018 | July 2018
  
  Publication
- EventWiki: A Knowledge Base of Major Events
  
  Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
  
  LREC 2018 | May 2018
  
  Publication Publication Publication
- Leveraging Contextual Sentence Relations for Extractive Summarization Using a Neural Attention Model
  
  Pengjie Ren, Zhumin Chen, Zhaochun Ren, Furu Wei, Jun Ma, Maarten de Rijke
  
  SIGIR '17, Shinjuku, Tokyo, Japan, August 07-11, 2017 | August 2017
  
  Publication
- SuperAgent: A Customer Service Chatbot for E-commerce Websites
  
  Lei Cui, Furu Wei, Shaohan Huang, Chuanqi Tan, Chaoqun Duan, Ming Zhou
  
  ACL 2017 | July 2017
  
  Publication Publication
- QuickView: advanced search of tweets
  
  Xiaohua Liu, Long Jiang, Furu Wei, Ming Zhou, QuickView Team Microsoft
  
  SIGIR (Demo paper) | January 2011
  
  ISBN: 978-1-4503-0757-4
  
  Publication Publication
- Context Preserving Dynamic Word Cloud Visualization
  
  Weiwei Cui, Yingcai Wu, Shixia Liu, Furu Wei, Michelle Zhou, Huamin Qu
  
  2010 IEEE Pacific Visualization Symposium (PacificVis) | March 2010
  
  Publication Publication
- Model as a Game: On Numerical and Spatial Consistency for Generative Games
  
  Jingye Chen, Yuzhong Zhao, Yupan Huang, Lei Cui, Li Dong, Tengchao Lv, Qifeng Chen, Furu Wei
  
  March 2025
  
  March 2025
  
  Publication
- ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation
  
  Zongyi Li, Shujie Hu, Shujie Liu, Long Zhou, Jeongsoo Choi, Lingwei Meng, Xun Guo, Jinyu Li, Hefei Ling, Furu Wei
  
  October 2024
  
  October 2024
  
  Publication
- Examining False Positives under Inference Scaling for Mathematical Reasoning
  
  Yu Wang, Nan Yang, Liang Wang, Furu Wei
  
  MSR-TR-2025-2 | February 2025
  
  Published by Microsoft
  
  Publication
- Differential Transformer
  
  Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, Furu Wei
  
  MSR-TR-2024-42 | October 2024
  
  Published by Microsoft
  
  Publication
- Multilingual E5 Text Embeddings: A Technical Report
  
  Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
  
  MSR-TR-2024-45 | February 2024
  
  Published by Microsoft
  
  DOI Publication Publication Github
- Inference with Reference: Lossless Acceleration of Large Language Models
  
  Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang (姜大昕), Linjun Yang, Rangan Majumder, Furu Wei
  
  MSR-TR-2023-17 | April 2023
  
  Published by Microsoft
  
  Publication
- REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models
  
  Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang
  
  December 2024
  
  Publication
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Boosting Text-to-Video Generative Model with MLLMs Feedback
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Kosmos-2.5: A Multimodal Literate Model
  
  Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
  
  September 2023
  
  Preprint Project
- Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
  
  Wenhui Wang, Hangbo Bao, Li Dong, Kriti Aggarwal, Saksham Singhal, Subhojit Som, Furu Wei, Johan Bjorck, Zhiliang Peng, Qiang Liu, Owais Khan Mohammed
  
  August 2022
  
  Publication
- THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
  
  Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang (姜大昕), Haoyi Zhou, Jianxin Li, Furu Wei
  
  May 2022
- Document AI: Benchmarks, Models and Applications
  
  Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
  
  November 2021
  
  Publication Project
- Multilingual Agreement for Multilingual Neural Machine Translation
  
  Jian Yang, Yuwei Yin, Shuming Ma, Haoyang Huang, Dongdong Zhang, Zhoujun Li, Furu Wei
  
  July 2021
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication Project
- VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization
  
  Tengchao Lv, Lei Cui, Momcilo Vasilijevic, Furu Wei
  
  June 2021
  
  Preprint
- LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
  
  Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
  
  April 2021
  
  Publication Project

Furu Wei

Publications

DocReward: A Document Reward Model for Structuring and Stylizing

Chain-of-Retrieval Augmented Generation

Native Hybrid Thinking Models

Scaling Laws of Synthetic Data for Language Models

Reward Reasoning Model

Chain-of-Reasoning: Towards Unified Mathematical Reasoning in Large Language Models via a Multi-Paradigm Perspective

Boosting Large Language Model for Speech Synthesis: An Empirical Study

Model as a Game: On Numerical and Spatial Consistency for Generative Games

Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning

Examining False Positives under Inference Scaling for Mathematical Reasoning

PEACE: Empowering Geologic Map Holistic Understanding with MLLMs

Imagine while Reasoning in Space: Multimodal Visualization-of-Thought

Bootstrap Your Own Context Length

REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models

MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark

BitNet a4.8: 4-bit Activations for 1-bit LLMs

Little Giants: Synthesizing High-Quality Embedding Data at Scale

1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs

Differential Transformer

ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation

Boosting Text-to-Video Generative Model with MLLMs Feedback

Multimodal Large Language Models Make Text-to-Image Generative Models Align Better

Q-Sparse: All Large Language Models can be Fully Sparsely-Activated

Autoregressive Speech Synthesis without Vector Quantization

VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment

VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers

Direct Preference Knowledge Distillation for Large Language Models

Instruction Pre-Training: Language Models are Supervised Multitask Learners

VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers

A whole-slide foundation model for digital pathology from real-world data

xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

You Only Cache Once: Decoder-Decoder Architectures for Language Models

Kosmos-G: Generating Images in Context with Multimodal Large Language Models

Multi-Head Mixture-of-Experts

LongEmbed: Extending Embedding Models for Long Context Retrieval

Mind’s Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models

Mixture of LoRA Experts

LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models

WavLLM: Towards Robust and Adaptive Speech Large Language Model

SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data

MathScale: Scaling Instruction Tuning for Mathematical Reasoning

ResLoRA: Identity Residual Mapping in Low-Rank Adaption

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

Towards Optimal Learning of Language Models

Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models

Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models

Generative Representational Instruction Tuning

Multilingual E5 Text Embeddings: A Technical Report

Text Diffusion with Reinforced Conditioning

Improving Text Embeddings with Large Language Models

When an Image is Worth 1,024 x 1,024 Words: A Case Study in Computational Pathology

Large Search Model: Redefining Search Stack in the Era of LLMs

BitNet: Scaling 1-bit Transformers for Large Language Models

Fine-Tuning LLaMA for Multi-Stage Text Retrieval

Augmenting Language Models with Long-Term Memory

PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training

Adapting Large Language Models via Reading Comprehension

Kosmos-2.5: A Multimodal Literate Model

Retentive Network: A Successor to Transformer for Large Language Models

Generative Retrieval for Conversational Question Answering

LongNet: Scaling Transformers to 1,000,000,000 Tokens

In-context Autoencoder for Context Compression in a Large Language Model

Learning to Retrieve In-Context Examples for Large Language Models

Knowledge Distillation of Large Language Models

Learning to Rank in Generative Retrieval

Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting

Multiview Identifiers Enhanced Generative Retrieval

TextDiffuser: Diffusion Models as Text Painters

VatLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning

Inference with Reference: Lossless Acceleration of Large Language Models

Language Is Not All You Need: Aligning Perception with Language Models

Query2doc: Query Expansion with Large Language Models

TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

Text Embeddings by Weakly-Supervised Contrastive Pre-training

Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning

DiT: Self-supervised Pre-training for Document Image Transformer