Shaohan Huang

Senior Researcher

论文与出版物

查看方式：

- DocReward: A Document Reward Model for Structuring and Stylizing
  
  Junpeng Liu, Yuzhong Zhao, Bowen Cao, Jiayu Ding, Yilin Jia, Tengchao Lv, Yupan Huang, Shaohan Huang, Nan Yang, Li Dong, Lei Cui, Tao Ge, Xun Wang, Huitian Jiao, Sun Mao, Fnu Kartik, Siqing Chen, Wai Lam, Furu Wei
  
  October 2025
  
  Publication
- Native Hybrid Thinking Models
  
  Lingjie Jiang, Xun Wu, Shaohan Huang, Qingxiu Dong, Zewen Chi, Li Dong, Xingxing Zhang, Tengchao Lv, Lei Cui, Furu Wei
  
  NeurIPS 2025 | October 2025
  
  Publication
- Reward Reasoning Model
  
  Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, Furu Wei
  
  NeurIPS 2025 | May 2025
  
  DOI Publication Publication
- REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models
  
  Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang
  
  December 2024
  
  Publication
- Boosting Text-to-Video Generative Model with MLLMs Feedback
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Instruction Pre-Training: Language Models are Supervised Multitask Learners
  
  Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei
  
  June 2024
  
  DOI Publication Publication
- MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
  
  Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
  
  May 2024
  
  DOI Publication Publication
- You Only Cache Once: Decoder-Decoder Architectures for Language Models
  
  Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei
  
  NeurIPS 2024 | May 2024
  
  DOI Publication Publication
- Kosmos-G: Generating Images in Context with Multimodal Large Language Models
  
  Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei
  
  ICLR 2024 | May 2024
  
  DOI Publication PDF
- Multi-Head Mixture-of-Experts
  
  Xun Wu, Shaohan Huang, Wenhui Wang, Furu Wei
  
  NeurIPS 2024 | April 2024
  
  DOI Publication Publication
- Mixture of LoRA Experts
  
  Shaohan Huang, Furu Wei
  
  ICLR 2024 | April 2024
  
  Publication
- ResLoRA: Identity Residual Mapping in Low-Rank Adaption
  
  Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  February 2024
  
  DOI Publication Publication
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
  
  Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
  
  Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- Text Diffusion with Reinforced Conditioning
  
  Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  AAAI 2024 | February 2024
  
  Publication
- BitNet: Scaling 1-bit Transformers for Large Language Models
  
  Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei
  
  October 2023
  
  DOI Publication
- Adapting Large Language Models via Reading Comprehension
  
  Daixuan Cheng, Shaohan Huang, Furu Wei
  
  ICLR 2024 | September 2023
  
  DOI
- Kosmos-2.5: A Multimodal Literate Model
  
  Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
  
  September 2023
  
  Preprint 项目
- Retentive Network: A Successor to Transformer for Large Language Models
  
  Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei
  
  August 2023
  
  Publication
- LongNet: Scaling Transformers to 1,000,000,000 Tokens
  
  Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei
  
  July 2023
  
  DOI PDF 视频
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
  
  Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang (姜大昕), Haoyi Zhou, Jianxin Li, Furu Wei
  
  May 2022
- Learning Music Sequence Representation from Text Supervision
  
  Tianyu Chen, Shuai Zhang, Haoyi Zhou, Jianxin Li, Yuan Xie, Shaohan Huang
  
  April 2022
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication 项目
- MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
  
  Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  EMNLP 2021 | October 2021
  
  Publication
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication 项目
- Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
  
  Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | July 2021
  
  Publication Github
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication 项目
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication 项目
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication 项目
- Consistency Regularization for Cross-Lingual Fine-Tuning
  
  Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
  
  Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
  
  Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | December 2020
  
  PDF Publication
- LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  
  Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
  
  KDD 2020 | August 2020
  
  PDF Publication Github 项目

- DocReward: A Document Reward Model for Structuring and Stylizing
  
  Junpeng Liu, Yuzhong Zhao, Bowen Cao, Jiayu Ding, Yilin Jia, Tengchao Lv, Yupan Huang, Shaohan Huang, Nan Yang, Li Dong, Lei Cui, Tao Ge, Xun Wang, Huitian Jiao, Sun Mao, Fnu Kartik, Siqing Chen, Wai Lam, Furu Wei
  
  October 2025
  
  Publication
- Native Hybrid Thinking Models
  
  Lingjie Jiang, Xun Wu, Shaohan Huang, Qingxiu Dong, Zewen Chi, Li Dong, Xingxing Zhang, Tengchao Lv, Lei Cui, Furu Wei
  
  NeurIPS 2025 | October 2025
  
  Publication
- Reward Reasoning Model
  
  Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, Furu Wei
  
  NeurIPS 2025 | May 2025
  
  DOI Publication Publication
- Boosting Text-to-Video Generative Model with MLLMs Feedback
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Instruction Pre-Training: Language Models are Supervised Multitask Learners
  
  Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei
  
  June 2024
  
  DOI Publication Publication
- MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
  
  Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
  
  May 2024
  
  DOI Publication Publication
- You Only Cache Once: Decoder-Decoder Architectures for Language Models
  
  Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei
  
  NeurIPS 2024 | May 2024
  
  DOI Publication Publication
- Kosmos-G: Generating Images in Context with Multimodal Large Language Models
  
  Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei
  
  ICLR 2024 | May 2024
  
  DOI Publication PDF
- Multi-Head Mixture-of-Experts
  
  Xun Wu, Shaohan Huang, Wenhui Wang, Furu Wei
  
  NeurIPS 2024 | April 2024
  
  DOI Publication Publication
- ResLoRA: Identity Residual Mapping in Low-Rank Adaption
  
  Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  February 2024
  
  DOI Publication Publication
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
  
  Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
  
  Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- Text Diffusion with Reinforced Conditioning
  
  Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  AAAI 2024 | February 2024
  
  Publication
- BitNet: Scaling 1-bit Transformers for Large Language Models
  
  Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei
  
  October 2023
  
  DOI Publication
- Adapting Large Language Models via Reading Comprehension
  
  Daixuan Cheng, Shaohan Huang, Furu Wei
  
  ICLR 2024 | September 2023
  
  DOI
- Kosmos-2.5: A Multimodal Literate Model
  
  Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
  
  September 2023
  
  Preprint 项目
- Retentive Network: A Successor to Transformer for Large Language Models
  
  Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei
  
  August 2023
  
  Publication
- LongNet: Scaling Transformers to 1,000,000,000 Tokens
  
  Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei
  
  July 2023
  
  DOI PDF 视频
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication 项目
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication 项目
- Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
  
  Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | July 2021
  
  Publication Github
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication 项目
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication 项目
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication 项目
- Consistency Regularization for Cross-Lingual Fine-Tuning
  
  Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
  
  Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
  
  Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | December 2020
  
  PDF Publication
- LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  
  Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
  
  KDD 2020 | August 2020
  
  PDF Publication Github 项目
- REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models
  
  Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang
  
  December 2024
  
  Publication
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Kosmos-2.5: A Multimodal Literate Model
  
  Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
  
  September 2023
  
  Preprint 项目
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- Mixture of LoRA Experts
  
  Shaohan Huang, Furu Wei
  
  ICLR 2024 | April 2024
  
  Publication
- Kosmos-2.5: A Multimodal Literate Model
  
  Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
  
  September 2023
  
  Preprint 项目
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
  
  Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang (姜大昕), Haoyi Zhou, Jianxin Li, Furu Wei
  
  May 2022
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication 项目
- MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
  
  Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  EMNLP 2021 | October 2021
  
  Publication
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication 项目
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication 项目
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication 项目
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication 项目
- Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
  
  Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  
  Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
  
  KDD 2020 | August 2020
  
  PDF Publication Github 项目
- Learning Music Sequence Representation from Text Supervision
  
  Tianyu Chen, Shuai Zhang, Haoyi Zhou, Jianxin Li, Yuan Xie, Shaohan Huang
  
  April 2022

- DocReward: A Document Reward Model for Structuring and Stylizing
  
  Junpeng Liu, Yuzhong Zhao, Bowen Cao, Jiayu Ding, Yilin Jia, Tengchao Lv, Yupan Huang, Shaohan Huang, Nan Yang, Li Dong, Lei Cui, Tao Ge, Xun Wang, Huitian Jiao, Sun Mao, Fnu Kartik, Siqing Chen, Wai Lam, Furu Wei
  
  October 2025
  
  Publication
- Instruction Pre-Training: Language Models are Supervised Multitask Learners
  
  Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei
  
  June 2024
  
  DOI Publication Publication
- MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
  
  Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
  
  May 2024
  
  DOI Publication Publication
- ResLoRA: Identity Residual Mapping in Low-Rank Adaption
  
  Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  February 2024
  
  DOI Publication Publication
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
  
  Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
  
  Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei
  
  February 2024
  
  Work in progress
  
  DOI Publication Publication
- BitNet: Scaling 1-bit Transformers for Large Language Models
  
  Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei
  
  October 2023
  
  DOI Publication
- Retentive Network: A Successor to Transformer for Large Language Models
  
  Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei
  
  August 2023
  
  Publication
- LongNet: Scaling Transformers to 1,000,000,000 Tokens
  
  Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei
  
  July 2023
  
  DOI PDF 视频
- Native Hybrid Thinking Models
  
  Lingjie Jiang, Xun Wu, Shaohan Huang, Qingxiu Dong, Zewen Chi, Li Dong, Xingxing Zhang, Tengchao Lv, Lei Cui, Furu Wei
  
  NeurIPS 2025 | October 2025
  
  Publication
- Reward Reasoning Model
  
  Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, Furu Wei
  
  NeurIPS 2025 | May 2025
  
  DOI Publication Publication
- You Only Cache Once: Decoder-Decoder Architectures for Language Models
  
  Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei
  
  NeurIPS 2024 | May 2024
  
  DOI Publication Publication
- Kosmos-G: Generating Images in Context with Multimodal Large Language Models
  
  Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei
  
  ICLR 2024 | May 2024
  
  DOI Publication PDF
- Multi-Head Mixture-of-Experts
  
  Xun Wu, Shaohan Huang, Wenhui Wang, Furu Wei
  
  NeurIPS 2024 | April 2024
  
  DOI Publication Publication
- Mixture of LoRA Experts
  
  Shaohan Huang, Furu Wei
  
  ICLR 2024 | April 2024
  
  Publication
- Text Diffusion with Reinforced Conditioning
  
  Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
  
  AAAI 2024 | February 2024
  
  Publication
- Adapting Large Language Models via Reading Comprehension
  
  Daixuan Cheng, Shaohan Huang, Furu Wei
  
  ICLR 2024 | September 2023
  
  DOI
- Language Is Not All You Need: Aligning Perception with Language Models
  
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
  
  NeurIPS 2023 | March 2023
  
  Publication
- Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
  
  ACL 2023 | October 2022
  
  Work in progress
  
  Publication
- Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
  
  Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  WMT | November 2021
  
  Publication Publication 项目
- MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
  
  Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  EMNLP 2021 | October 2021
  
  Publication
- Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
  
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  EMNLP 2021 | September 2021
  
  Publication 项目
- Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
  
  Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | July 2021
  
  Publication Github
- XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
  
  ACL 2022 | June 2021
  
  Publication 项目
- Consistency Regularization for Cross-Lingual Fine-Tuning
  
  Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  PDF
- Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
  
  Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
  
  ACL-IJCNLP 2021 | June 2021
  
  Publication
- MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
  
  Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei
  
  ACL-IJCNLP 2021 | December 2020
  
  PDF Publication
- LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  
  Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
  
  KDD 2020 | August 2020
  
  PDF Publication Github 项目
- REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models
  
  Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang
  
  December 2024
  
  Publication
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Boosting Text-to-Video Generative Model with MLLMs Feedback
  
  Xun Wu, Shaohan Huang, Furu Wei
  
  October 2024
  
  Publication
- Kosmos-2.5: A Multimodal Literate Model
  
  Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
  
  September 2023
  
  Preprint 项目
- THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
  
  Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang (姜大昕), Haoyi Zhou, Jianxin Li, Furu Wei
  
  May 2022
- Learning Music Sequence Representation from Text Supervision
  
  Tianyu Chen, Shuai Zhang, Haoyi Zhou, Jianxin Li, Yuan Xie, Shaohan Huang
  
  April 2022
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  June 2021
  
  Publication Publication 项目
- DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
  
  Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
  
  arXiv: Computation and Language | June 2021
  
  Publication Publication Publication 项目

Shaohan Huang

论文与出版物

DocReward: A Document Reward Model for Structuring and Stylizing

Native Hybrid Thinking Models

Reward Reasoning Model

REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models

Boosting Text-to-Video Generative Model with MLLMs Feedback

Multimodal Large Language Models Make Text-to-Image Generative Models Align Better

Instruction Pre-Training: Language Models are Supervised Multitask Learners

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

You Only Cache Once: Decoder-Decoder Architectures for Language Models

Kosmos-G: Generating Images in Context with Multimodal Large Language Models

Multi-Head Mixture-of-Experts

Mixture of LoRA Experts

ResLoRA: Identity Residual Mapping in Low-Rank Adaption

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models

Text Diffusion with Reinforced Conditioning

BitNet: Scaling 1-bit Transformers for Large Language Models

Adapting Large Language Models via Reading Comprehension

Kosmos-2.5: A Multimodal Literate Model

Retentive Network: A Successor to Transformer for Large Language Models

LongNet: Scaling Transformers to 1,000,000,000 Tokens

Language Is Not All You Need: Aligning Perception with Language Models

Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning

THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption

Learning Music Sequence Representation from Text Supervision

Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.

MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.

Consistency Regularization for Cross-Lingual Fine-Tuning

Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment

MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

LayoutLM: Pre-training of Text and Layout for Document Image Understanding

DocReward: A Document Reward Model for Structuring and Stylizing

Native Hybrid Thinking Models

Reward Reasoning Model

Boosting Text-to-Video Generative Model with MLLMs Feedback

Instruction Pre-Training: Language Models are Supervised Multitask Learners

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

You Only Cache Once: Decoder-Decoder Architectures for Language Models

Kosmos-G: Generating Images in Context with Multimodal Large Language Models

Multi-Head Mixture-of-Experts

ResLoRA: Identity Residual Mapping in Low-Rank Adaption

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models

Text Diffusion with Reinforced Conditioning

BitNet: Scaling 1-bit Transformers for Large Language Models

Adapting Large Language Models via Reading Comprehension

Kosmos-2.5: A Multimodal Literate Model

Retentive Network: A Successor to Transformer for Large Language Models

LongNet: Scaling Transformers to 1,000,000,000 Tokens

Language Is Not All You Need: Aligning Perception with Language Models

Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning

Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.

Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.

Consistency Regularization for Cross-Lingual Fine-Tuning

Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment

MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

LayoutLM: Pre-training of Text and Layout for Document Image Understanding

REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models

Multimodal Large Language Models Make Text-to-Image Generative Models Align Better

Kosmos-2.5: A Multimodal Literate Model

Language Is Not All You Need: Aligning Perception with Language Models

Mixture of LoRA Experts

Kosmos-2.5: A Multimodal Literate Model

Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning

THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption

Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.

MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA