Sheng Zhao

Partner Group Engineer Manager

Publications

View by:

- FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
  
  Jiaqi Li, Yao Qian, Yuxuan Hu, Leying Zhang, Xiaofei Wang, Heng Lu, Manthan Thakker, Jinyu Li, Sheng Zhao, Zhizheng Wu
  
  ICLR 2026 | September 2025
  
  DOI Publication
- CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching
  
  Leying Zhang, Yao Qian, Xiaofei Wang, Manthan Thakker, Dongmei Wang, Jianwei Yu, Haibin Wu, Yuxuan Hu, Jinyu Li, Yanmin Qian, Sheng Zhao
  
  NeurIPS 2025 | May 2025
  
  DOI
- Total-Duration-Aware Duration Modeling for Text-to-Speech Systems
  
  Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Chung-Hsien Tsai, Canrun Li, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Jinyu Li, Sheng Zhao, Naoyuki Kanda
  
  Interspeech 2024 | September 2024
  
  Publication
- Autoregressive Speech Synthesis without Vector Quantization
  
  Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, Furu Wei
  
  July 2024
  
  Publication Project
- VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
  
  Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, Furu Wei
  
  June 2024
  
  Publication Project
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
  
  Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
  
  June 2024
  
  Publication Project
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
  
  Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
  
  June 2024
  
  DOI Publication Publication Project
- Memories are One-to-Many Mapping Alleviators in Talking Face Generation
  
  Tianyu He, Sheng Zhao, Jiang Bian
  
  IEEE Transactions on Pattern Analysis and Machine Intelligence | June 2024
  
  Publication
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
  
  Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiang-Yang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, Sheng Zhao
  
  ICML | June 2024
  
  Publication
- TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation
  
  Chenyang Le, Yao Qian, Dongmei Wang, Long Zhou, Shujie Liu, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Sheng Zhao, Michael Zeng
  
  NeurIPS 2024 | May 2024
  
  Publication Project
- GAIA: Zero-shot Talking Avatar Generation
  
  Tianyu He, Junliang Guo, Runyi Yu, Yuchi Wang, Jialiang Zhu, Kaikai An, Leyi Li, Xu Tan, Chunyu Wang, HsiangTao Wu, Sheng Zhao, Jiang Bian
  
  2024 International Conference on Learning Representations | May 2024
  
  DOI Publication
- CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations
  
  Leying Zhang, Yao Qian, Long Zhou, Shujie Liu (shujliu), Dongmei Wang, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Lei He, Sheng Zhao, Michael Zeng
  
  NeurIPS 2024 | April 2024
  
  DOI Publication Project
- AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models
  
  Yuancheng Wang, Zeqian Ju, Xu Tan, Lei He, Zhizheng Wu, Jiang Bian, Sheng Zhao
  
  NeurIPS 2023 | December 2023
  
  Publication
- DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder
  
  Tianyu He, Sheng Zhao, Jiang Bian
  
  ACM International Conference on Multimedia | October 2023
  
  Publication
- HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details
  
  Tianyu He, Muscle Wu, Sheng Zhao, Jiang Bian
  
  2023 International Conference on Computer Vision | October 2023
  
  Publication
- Large-Scale Automatic Audiobook Creation
  
  Brendan Walsh, Mark Hamilton, Greg Newby, Xi Wang, Serena Ruan, Sheng Zhao, Lei He, Shaofei Zhang, Eric Dettinger, William T. Freeman, Markus Weimer
  
  Interspeech Show and Tell | September 2023
  
  Publication Project
- PromptTTS 2: Describing and Generating Voices with Text Prompt
  
  Yichong Leng, Zhifang Guo, Kai Shen, Xu Tan, Zeqian Ju, Yanqing Liu, Yufei Liu, Dongchao Yang, Leying Zhang, Kaitao Song, Lei He, Xiang-Yang Li, Sheng Zhao, Tao Qin, Jiang Bian
  
  ICLR 2024 | September 2023
  
  DOI PDF
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
  
  Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, Jiang Bian
  
  ICLR 2024 | April 2023
  
  DOI PDF
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
  
  Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, Furu Wei
  
  January 2023
  
  Publication PDF Project
- RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion
  
  Dacheng Yin, Chuanxin Tang, Yanqing Liu, Xiaoqiang Wang, Zhiyuan Zhao, Yucheng Zhao, Zhiwei Xiong, Sheng Zhao, Chong Luo
  
  Interspeech 2022 | July 2022
  
  PrePrint
  
  Publication
- MeloForm: Generating Melody with Musical Form based on Expert Systems and Neural Networks
  
  Peiling Lu, Xu Tan, Botao Yu, Tao Qin, Sheng Zhao, Tie-Yan Liu
  
  June 2022
  
  Publication
- Mixed-phoneme bert: Improving bert with mixed phoneme and sup-phoneme representations for text to speech
  
  Guangyan Zhang, Kaitao Song, Xu Tan, Daxin Tan, Yuzi Yan, Yanqing Liu, Gang Wang, Wei Zhou, Tao Qin, Tan Lee, Sheng Zhao
  
  INTERSPEECH 2022 | June 2022
  
  Publication
- InferGrad: Improving Diffusion Models for Vocoder by Considering Inference in Training
  
  Zehua Chen, Xu Tan, Ke Wang, Shifeng Pan, Danilo Mandic, Lei He, Sheng Zhao
  
  ICASSP 2022 | February 2022
- FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
  
  Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu
  
  ICLR 2021 | December 2021
  
  Publication
- DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021
  
  Sheng Zhao
  
  November 2021
  
  Publication

- FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
  
  Jiaqi Li, Yao Qian, Yuxuan Hu, Leying Zhang, Xiaofei Wang, Heng Lu, Manthan Thakker, Jinyu Li, Sheng Zhao, Zhizheng Wu
  
  ICLR 2026 | September 2025
  
  DOI Publication
- CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching
  
  Leying Zhang, Yao Qian, Xiaofei Wang, Manthan Thakker, Dongmei Wang, Jianwei Yu, Haibin Wu, Yuxuan Hu, Jinyu Li, Yanmin Qian, Sheng Zhao
  
  NeurIPS 2025 | May 2025
  
  DOI
- Total-Duration-Aware Duration Modeling for Text-to-Speech Systems
  
  Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Chung-Hsien Tsai, Canrun Li, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Jinyu Li, Sheng Zhao, Naoyuki Kanda
  
  Interspeech 2024 | September 2024
  
  Publication
- Autoregressive Speech Synthesis without Vector Quantization
  
  Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, Furu Wei
  
  July 2024
  
  Publication Project
- VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
  
  Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, Furu Wei
  
  June 2024
  
  Publication Project
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
  
  Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
  
  June 2024
  
  DOI Publication Publication Project
- Memories are One-to-Many Mapping Alleviators in Talking Face Generation
  
  Tianyu He, Sheng Zhao, Jiang Bian
  
  IEEE Transactions on Pattern Analysis and Machine Intelligence | June 2024
  
  Publication
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
  
  Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiang-Yang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, Sheng Zhao
  
  ICML | June 2024
  
  Publication
- TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation
  
  Chenyang Le, Yao Qian, Dongmei Wang, Long Zhou, Shujie Liu, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Sheng Zhao, Michael Zeng
  
  NeurIPS 2024 | May 2024
  
  Publication Project
- GAIA: Zero-shot Talking Avatar Generation
  
  Tianyu He, Junliang Guo, Runyi Yu, Yuchi Wang, Jialiang Zhu, Kaikai An, Leyi Li, Xu Tan, Chunyu Wang, HsiangTao Wu, Sheng Zhao, Jiang Bian
  
  2024 International Conference on Learning Representations | May 2024
  
  DOI Publication
- CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations
  
  Leying Zhang, Yao Qian, Long Zhou, Shujie Liu (shujliu), Dongmei Wang, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Lei He, Sheng Zhao, Michael Zeng
  
  NeurIPS 2024 | April 2024
  
  DOI Publication Project
- AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models
  
  Yuancheng Wang, Zeqian Ju, Xu Tan, Lei He, Zhizheng Wu, Jiang Bian, Sheng Zhao
  
  NeurIPS 2023 | December 2023
  
  Publication
- DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder
  
  Tianyu He, Sheng Zhao, Jiang Bian
  
  ACM International Conference on Multimedia | October 2023
  
  Publication
- HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details
  
  Tianyu He, Muscle Wu, Sheng Zhao, Jiang Bian
  
  2023 International Conference on Computer Vision | October 2023
  
  Publication
- Large-Scale Automatic Audiobook Creation
  
  Brendan Walsh, Mark Hamilton, Greg Newby, Xi Wang, Serena Ruan, Sheng Zhao, Lei He, Shaofei Zhang, Eric Dettinger, William T. Freeman, Markus Weimer
  
  Interspeech Show and Tell | September 2023
  
  Publication Project
- PromptTTS 2: Describing and Generating Voices with Text Prompt
  
  Yichong Leng, Zhifang Guo, Kai Shen, Xu Tan, Zeqian Ju, Yanqing Liu, Yufei Liu, Dongchao Yang, Leying Zhang, Kaitao Song, Lei He, Xiang-Yang Li, Sheng Zhao, Tao Qin, Jiang Bian
  
  ICLR 2024 | September 2023
  
  DOI PDF
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
  
  Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, Jiang Bian
  
  ICLR 2024 | April 2023
  
  DOI PDF
- MeloForm: Generating Melody with Musical Form based on Expert Systems and Neural Networks
  
  Peiling Lu, Xu Tan, Botao Yu, Tao Qin, Sheng Zhao, Tie-Yan Liu
  
  June 2022
  
  Publication
- FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
  
  Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu
  
  ICLR 2021 | December 2021
  
  Publication
- DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021
  
  Sheng Zhao
  
  November 2021
  
  Publication
- FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
  
  Jiaqi Li, Yao Qian, Yuxuan Hu, Leying Zhang, Xiaofei Wang, Heng Lu, Manthan Thakker, Jinyu Li, Sheng Zhao, Zhizheng Wu
  
  ICLR 2026 | September 2025
  
  DOI Publication
- CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching
  
  Leying Zhang, Yao Qian, Xiaofei Wang, Manthan Thakker, Dongmei Wang, Jianwei Yu, Haibin Wu, Yuxuan Hu, Jinyu Li, Yanmin Qian, Sheng Zhao
  
  NeurIPS 2025 | May 2025
  
  DOI
- Total-Duration-Aware Duration Modeling for Text-to-Speech Systems
  
  Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Chung-Hsien Tsai, Canrun Li, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Jinyu Li, Sheng Zhao, Naoyuki Kanda
  
  Interspeech 2024 | September 2024
  
  Publication
- Autoregressive Speech Synthesis without Vector Quantization
  
  Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, Furu Wei
  
  July 2024
  
  Publication Project
- VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
  
  Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, Furu Wei
  
  June 2024
  
  Publication Project
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
  
  Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
  
  June 2024
  
  DOI Publication Publication Project
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
  
  Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiang-Yang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, Sheng Zhao
  
  ICML | June 2024
  
  Publication
- TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation
  
  Chenyang Le, Yao Qian, Dongmei Wang, Long Zhou, Shujie Liu, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Sheng Zhao, Michael Zeng
  
  NeurIPS 2024 | May 2024
  
  Publication Project
- CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations
  
  Leying Zhang, Yao Qian, Long Zhou, Shujie Liu (shujliu), Dongmei Wang, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Lei He, Sheng Zhao, Michael Zeng
  
  NeurIPS 2024 | April 2024
  
  DOI Publication Project
- Large-Scale Automatic Audiobook Creation
  
  Brendan Walsh, Mark Hamilton, Greg Newby, Xi Wang, Serena Ruan, Sheng Zhao, Lei He, Shaofei Zhang, Eric Dettinger, William T. Freeman, Markus Weimer
  
  Interspeech Show and Tell | September 2023
  
  Publication Project
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
  
  Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, Furu Wei
  
  January 2023
  
  Publication PDF Project
- RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion
  
  Dacheng Yin, Chuanxin Tang, Yanqing Liu, Xiaoqiang Wang, Zhiyuan Zhao, Yucheng Zhao, Zhiwei Xiong, Sheng Zhao, Chong Luo
  
  Interspeech 2022 | July 2022
  
  PrePrint
  
  Publication
- MeloForm: Generating Melody with Musical Form based on Expert Systems and Neural Networks
  
  Peiling Lu, Xu Tan, Botao Yu, Tao Qin, Sheng Zhao, Tie-Yan Liu
  
  June 2022
  
  Publication
- Mixed-phoneme bert: Improving bert with mixed phoneme and sup-phoneme representations for text to speech
  
  Guangyan Zhang, Kaitao Song, Xu Tan, Daxin Tan, Yuzi Yan, Yanqing Liu, Gang Wang, Wei Zhou, Tao Qin, Tan Lee, Sheng Zhao
  
  INTERSPEECH 2022 | June 2022
  
  Publication
- InferGrad: Improving Diffusion Models for Vocoder by Considering Inference in Training
  
  Zehua Chen, Xu Tan, Ke Wang, Shifeng Pan, Danilo Mandic, Lei He, Sheng Zhao
  
  ICASSP 2022 | February 2022
- FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
  
  Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu
  
  ICLR 2021 | December 2021
  
  Publication
- Memories are One-to-Many Mapping Alleviators in Talking Face Generation
  
  Tianyu He, Sheng Zhao, Jiang Bian
  
  IEEE Transactions on Pattern Analysis and Machine Intelligence | June 2024
  
  Publication
- DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder
  
  Tianyu He, Sheng Zhao, Jiang Bian
  
  ACM International Conference on Multimedia | October 2023
  
  Publication
- HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details
  
  Tianyu He, Muscle Wu, Sheng Zhao, Jiang Bian
  
  2023 International Conference on Computer Vision | October 2023
  
  Publication
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
  
  Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
  
  June 2024
  
  Publication Project
- PromptTTS 2: Describing and Generating Voices with Text Prompt
  
  Yichong Leng, Zhifang Guo, Kai Shen, Xu Tan, Zeqian Ju, Yanqing Liu, Yufei Liu, Dongchao Yang, Leying Zhang, Kaitao Song, Lei He, Xiang-Yang Li, Sheng Zhao, Tao Qin, Jiang Bian
  
  ICLR 2024 | September 2023
  
  DOI PDF
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
  
  Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, Jiang Bian
  
  ICLR 2024 | April 2023
  
  DOI PDF
- RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion
  
  Dacheng Yin, Chuanxin Tang, Yanqing Liu, Xiaoqiang Wang, Zhiyuan Zhao, Yucheng Zhao, Zhiwei Xiong, Sheng Zhao, Chong Luo
  
  Interspeech 2022 | July 2022
  
  PrePrint
  
  Publication
- FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
  
  Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu
  
  ICLR 2021 | December 2021
  
  Publication

- FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
  
  Jiaqi Li, Yao Qian, Yuxuan Hu, Leying Zhang, Xiaofei Wang, Heng Lu, Manthan Thakker, Jinyu Li, Sheng Zhao, Zhizheng Wu
  
  ICLR 2026 | September 2025
  
  DOI Publication
- CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching
  
  Leying Zhang, Yao Qian, Xiaofei Wang, Manthan Thakker, Dongmei Wang, Jianwei Yu, Haibin Wu, Yuxuan Hu, Jinyu Li, Yanmin Qian, Sheng Zhao
  
  NeurIPS 2025 | May 2025
  
  DOI
- Total-Duration-Aware Duration Modeling for Text-to-Speech Systems
  
  Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Chung-Hsien Tsai, Canrun Li, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Jinyu Li, Sheng Zhao, Naoyuki Kanda
  
  Interspeech 2024 | September 2024
  
  Publication
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
  
  Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiang-Yang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, Sheng Zhao
  
  ICML | June 2024
  
  Publication
- TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation
  
  Chenyang Le, Yao Qian, Dongmei Wang, Long Zhou, Shujie Liu, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Sheng Zhao, Michael Zeng
  
  NeurIPS 2024 | May 2024
  
  Publication Project
- GAIA: Zero-shot Talking Avatar Generation
  
  Tianyu He, Junliang Guo, Runyi Yu, Yuchi Wang, Jialiang Zhu, Kaikai An, Leyi Li, Xu Tan, Chunyu Wang, HsiangTao Wu, Sheng Zhao, Jiang Bian
  
  2024 International Conference on Learning Representations | May 2024
  
  DOI Publication
- CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations
  
  Leying Zhang, Yao Qian, Long Zhou, Shujie Liu (shujliu), Dongmei Wang, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Lei He, Sheng Zhao, Michael Zeng
  
  NeurIPS 2024 | April 2024
  
  DOI Publication Project
- AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models
  
  Yuancheng Wang, Zeqian Ju, Xu Tan, Lei He, Zhizheng Wu, Jiang Bian, Sheng Zhao
  
  NeurIPS 2023 | December 2023
  
  Publication
- DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder
  
  Tianyu He, Sheng Zhao, Jiang Bian
  
  ACM International Conference on Multimedia | October 2023
  
  Publication
- HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details
  
  Tianyu He, Muscle Wu, Sheng Zhao, Jiang Bian
  
  2023 International Conference on Computer Vision | October 2023
  
  Publication
- Large-Scale Automatic Audiobook Creation
  
  Brendan Walsh, Mark Hamilton, Greg Newby, Xi Wang, Serena Ruan, Sheng Zhao, Lei He, Shaofei Zhang, Eric Dettinger, William T. Freeman, Markus Weimer
  
  Interspeech Show and Tell | September 2023
  
  Publication Project
- PromptTTS 2: Describing and Generating Voices with Text Prompt
  
  Yichong Leng, Zhifang Guo, Kai Shen, Xu Tan, Zeqian Ju, Yanqing Liu, Yufei Liu, Dongchao Yang, Leying Zhang, Kaitao Song, Lei He, Xiang-Yang Li, Sheng Zhao, Tao Qin, Jiang Bian
  
  ICLR 2024 | September 2023
  
  DOI PDF
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
  
  Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, Jiang Bian
  
  ICLR 2024 | April 2023
  
  DOI PDF
- RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion
  
  Dacheng Yin, Chuanxin Tang, Yanqing Liu, Xiaoqiang Wang, Zhiyuan Zhao, Yucheng Zhao, Zhiwei Xiong, Sheng Zhao, Chong Luo
  
  Interspeech 2022 | July 2022
  
  PrePrint
  
  Publication
- MeloForm: Generating Melody with Musical Form based on Expert Systems and Neural Networks
  
  Peiling Lu, Xu Tan, Botao Yu, Tao Qin, Sheng Zhao, Tie-Yan Liu
  
  June 2022
  
  Publication
- Mixed-phoneme bert: Improving bert with mixed phoneme and sup-phoneme representations for text to speech
  
  Guangyan Zhang, Kaitao Song, Xu Tan, Daxin Tan, Yuzi Yan, Yanqing Liu, Gang Wang, Wei Zhou, Tao Qin, Tan Lee, Sheng Zhao
  
  INTERSPEECH 2022 | June 2022
  
  Publication
- InferGrad: Improving Diffusion Models for Vocoder by Considering Inference in Training
  
  Zehua Chen, Xu Tan, Ke Wang, Shifeng Pan, Danilo Mandic, Lei He, Sheng Zhao
  
  ICASSP 2022 | February 2022
- FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
  
  Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu
  
  ICLR 2021 | December 2021
  
  Publication
- Autoregressive Speech Synthesis without Vector Quantization
  
  Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, Furu Wei
  
  July 2024
  
  Publication Project
- VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
  
  Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, Furu Wei
  
  June 2024
  
  Publication Project
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
  
  Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
  
  June 2024
  
  Publication Project
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
  
  Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
  
  June 2024
  
  DOI Publication Publication Project
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
  
  Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, Furu Wei
  
  January 2023
  
  Publication PDF Project
- DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021
  
  Sheng Zhao
  
  November 2021
  
  Publication
- Memories are One-to-Many Mapping Alleviators in Talking Face Generation
  
  Tianyu He, Sheng Zhao, Jiang Bian
  
  IEEE Transactions on Pattern Analysis and Machine Intelligence | June 2024
  
  Publication

Sheng Zhao

Publications

FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates

CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching

Total-Duration-Aware Duration Modeling for Text-to-Speech Systems

Autoregressive Speech Synthesis without Vector Quantization

VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment

VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers

VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers

Memories are One-to-Many Mapping Alleviators in Talking Face Generation

NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models

TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation

GAIA: Zero-shot Talking Avatar Generation

CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations

AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models

DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder

HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details

Large-Scale Automatic Audiobook Creation

PromptTTS 2: Describing and Generating Voices with Text Prompt

NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion

MeloForm: Generating Melody with Musical Form based on Expert Systems and Neural Networks

Mixed-phoneme bert: Improving bert with mixed phoneme and sup-phoneme representations for text to speech

InferGrad: Improving Diffusion Models for Vocoder by Considering Inference in Training

FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021

FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates

CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching

Total-Duration-Aware Duration Modeling for Text-to-Speech Systems

Autoregressive Speech Synthesis without Vector Quantization

VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment

VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers

Memories are One-to-Many Mapping Alleviators in Talking Face Generation

NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models

TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation

GAIA: Zero-shot Talking Avatar Generation

CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations

AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models

DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder

HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details

Large-Scale Automatic Audiobook Creation

PromptTTS 2: Describing and Generating Voices with Text Prompt

NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers

MeloForm: Generating Melody with Musical Form based on Expert Systems and Neural Networks

FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021

FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates

CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching

Total-Duration-Aware Duration Modeling for Text-to-Speech Systems

Autoregressive Speech Synthesis without Vector Quantization

VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment

VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers

NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models

TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation

CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations

Large-Scale Automatic Audiobook Creation

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion

MeloForm: Generating Melody with Musical Form based on Expert Systems and Neural Networks

Mixed-phoneme bert: Improving bert with mixed phoneme and sup-phoneme representations for text to speech

InferGrad: Improving Diffusion Models for Vocoder by Considering Inference in Training

FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

Memories are One-to-Many Mapping Alleviators in Talking Face Generation

DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder

HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details

VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers

PromptTTS 2: Describing and Generating Voices with Text Prompt

NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers

RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion

FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates

CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching

Total-Duration-Aware Duration Modeling for Text-to-Speech Systems

NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models

TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation

GAIA: Zero-shot Talking Avatar Generation

CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations

AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models

DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder