{"id":804160,"date":"2021-12-17T15:22:31","date_gmt":"2021-12-17T23:22:31","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=804160"},"modified":"2021-12-20T07:18:37","modified_gmt":"2021-12-20T15:18:37","slug":"azure-ai-milestone-new-neural-text-to-speech-models-more-closely-mirror-natural-speech","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/azure-ai-milestone-new-neural-text-to-speech-models-more-closely-mirror-natural-speech\/","title":{"rendered":"Azure AI milestone: New Neural Text-to-Speech models more closely mirror natural speech"},"content":{"rendered":"\n
\"diagram\"<\/figure>\n\n\n\n

Neural Text-to-Speech\u2014along with recent milestones in computer vision<\/a> and question answering\u2014is part of a larger Azure AI (opens in new tab)<\/span><\/a> mission to provide relevant, meaningful AI solutions and services that work better for people because they better capture how people learn and work\u2014with improved vision, knowledge understanding, and speech capabilities. At the center of these efforts is XYZ-code, a joint representation of three cognitive attributes: monolingual text (X), audio or visual sensory signals (Y), and multilingual (Z). For more information about these efforts, read the <\/em>XYZ-code blog post<\/em><\/a>. <\/em><\/p>\n\n\n\n

Neural Text-to-Speech (opens in new tab)<\/span><\/a>\u202f(Neural TTS), a powerful speech synthesis capability of Azure Cognitive Services (opens in new tab)<\/span><\/a>, enables developers to convert text to lifelike speech. It is used in voice assistant scenarios, content read aloud capabilities, accessibility tools, and more. Neural TTS has now reached a significant milestone in Azure, with a new generation of Neural TTS model called Uni-TTSv4, whose quality shows no significant difference from sentence-level natural speech recordings.  <\/p>\n\n\n\n

Microsoft debuted the original technology three years ago, with close to human-parity (opens in new tab)<\/span><\/a> quality. This resulted in TTS audio that was more fluid, natural sounding, and better articulated. Since then, Neural TTS has been incorporated into Microsoft flagship products such as Edge Read Aloud (opens in new tab)<\/span><\/a>, Immersive Reader (opens in new tab)<\/span><\/a>, and Word Read Aloud (opens in new tab)<\/span><\/a>. It\u2019s also been adopted by many customers such as AT&T (opens in new tab)<\/span><\/a>, Duolingo (opens in new tab)<\/span><\/a>, Progressive (opens in new tab)<\/span><\/a>, and more. Users can choose from multiple pre-set voices or record and upload their own sample to create custom voices instead. Over 110 languages are supported, including a wide array of language variants, also known as locales.  <\/p>\n\n\n\n

The latest version of the model, Uni-TTSv4, is now shipping into production on a first set of eight voices (shown in the table below). We will continue to roll\u202fout the\u202fnew model\u202farchitecture to the remaining 110-plus languages\u202fand Custom Neural Voice (opens in new tab)<\/span><\/a> in the coming milestone. Our users will automatically get significantly better-quality\u202fTTS through the Azure TTS API (opens in new tab)<\/span><\/a>,\u202fMicrosoft Office,\u202fand\u202fEdge browser.\u202f<\/p>\n\n\n\n

Measuring TTS quality<\/h2>\n\n\n\n

Text-to-speech quality is measured by the Mean Opinion Score (MOS), a widely recognized scoring method for speech quality evaluation. For MOS studies, participants rate speech characteristics for both recordings of peoples\u2019 voices and TTS voices on a five-point scale. These characteristics include sound quality, pronunciation, speaking rate, and articulation. For any model improvement, we first conduct a side-by-side comparative MOS test (CMOS (opens in new tab)<\/span><\/a>) with production models. Then, we do a blind MOS test on the held-out recording set (recordings not used in training) and the TTS-synthesized audio and measure the difference between the two MOS scores. <\/p>\n\n\n\n

During research of the new model, Microsoft submitted the Uni-TTSv4 system to Blizzard Challenge 2021 (opens in new tab)<\/span><\/a> under its code name, DelightfulTTS. Our paper, \u201cDelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021<\/a>,\u201d provides in-depth detail of our research and the results. The Blizzard Challenge is a well-known TTS benchmark organized by world-class experts in TTS fields, and it conducts large-scale MOS tests on multiple TTS systems with hundreds of listeners. Results from Blizzard Challenge 2021 demonstrate that the voice built with the new model shows no significant difference from natural speech on the common dataset.<\/p>\n\n\n\n\n\t

\n\t\t\n\n\t\t

\n\t\tPODCAST SERIES<\/span>\n\t<\/p>\n\t\n\t

\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"Illustrated\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

AI Testing and Evaluation: Learnings from Science and Industry<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

Discover how Microsoft is learning from other domains to advance evaluation and testing as a pillar of AI governance.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t
\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tListen now\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t<\/div>\n\t<\/div>\n\t\n\n\n\n

Measurement results for Uni-TTSv4 and comparison<\/h2>\n\n\n\n

The MOS scores below are based on samples produced by the Uni-TTSv4 model under the constraints of real-time performance requirements.<\/p>\n\n\n\n

A Wilcoxon signed-rank test (opens in new tab)<\/span><\/a> was used to determine whether the MOS scores differed significantly between the held-out recordings and TTS. A\u202fp-value less than 0.05 (typically \u2264 0.05) is statistically significant and a\u202fp-value higher than 0.05 (> 0.05) is not statistically significant. A positive CMOS number shows gain over production, which shows it is more highly preferred by people judging the voice in terms of naturalness. <\/p>\n\n\n\n

Locale (voice)<\/th>Human recording (MOS)<\/th>Uni-TTSv4 (MOS)<\/th>Wilcoxon p-value<\/th>CMOS vs PROD<\/th><\/tr><\/thead>
En-US (Jenny) <\/td>4.33(\u00b10.04) <\/td>4.29(\u00b10.04) <\/td>0.266 <\/td>+0.116 <\/td><\/tr>
En-US (Sara)<\/td>4.16(\u00b10.05) <\/td>4.12 (\u00b10.05)<\/td>0.41 <\/td>+0.129 <\/td><\/tr>
Zh-CN (Xiaoxiao)<\/td>4.54(\u00b10.05) <\/td>4.51(\u00b10.05) <\/td>0.44 <\/td>+0.181 <\/td><\/tr>
It-IT (Elsa) <\/td>4.59(\u00b10.04) <\/td>4.58(\u00b10.03) <\/td>0.34 <\/td>+0.25 <\/td><\/tr>
Ja-JP (Nanami) <\/td>4.44(\u00b10.04) <\/td>4.37(\u00b10.05) <\/td>0.053 <\/td>+0.19 <\/td><\/tr>
Ko-KR(Sun-hi) <\/td>4.24(\u00b10.06) <\/td>4.15(\u00b10.06) <\/td>0.11 <\/td>+0.097 <\/td><\/tr>
Es-ES (Alvaro) <\/td>4.36(\u00b10.05) <\/td>4.33(\u00b10.04) <\/td>0.312 <\/td>+0.18 <\/td><\/tr>
Es-MX (Dalia) <\/td>4.45 (\u00b10.05) <\/td>4.39(\u00b10.05) <\/td>0.103 <\/td>+0.076 <\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

A comparison of human and Uni-TTSv4 audio samples <\/h2>\n\n\n\n

Listen to the recording and TTS samples below to hear the quality of the new model. Note that the recording is not part of the training set.<\/p>\n\n\n\n

These voices are updated to the new model in the Azure TTS online service. You can also try the demo (opens in new tab)<\/span><\/a> with your own text. More voices will be upgraded to Uni-TTSv4 later.<\/p>\n\n\n\n

<\/div>\n\n\n\n
\n
\n

En-US (Jenny)<\/strong><\/p>\n\n\n\n

The visualizations of the vocal quality continue in a quartet and octet.<\/p>\n\n\n\n

<\/p>\n<\/div>\n\n\n\n

\n

Human recording<\/strong><\/p>\n\n\n\n

\n

Uni-TTSv4<\/strong><\/p>\n\n\n\n

\n
\n

En-US (Sara)<\/strong><\/p>\n\n\n\n

Like other visitors, he is a believer.<\/p>\n\n\n\n

<\/p>\n<\/div>\n\n\n\n

\n

Human recording<\/strong><\/p>\n\n\n\n