{"id":444522,"date":"2017-12-06T12:45:40","date_gmt":"2017-12-06T20:45:40","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=444522"},"modified":"2017-12-07T09:19:15","modified_gmt":"2017-12-07T17:19:15","slug":"deliberation-networks","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/deliberation-networks\/","title":{"rendered":"Deliberation Network: Pushing the frontiers of neural machine translation"},"content":{"rendered":"

During the Tang dynasty of China, which lasted from 618 to 907, the poet Jia Dao was known for polishing his poems over and over to make them better and better. One famous story describes how he deliberated over two lines of a poem that read, \u201cBirds nestle in the trees by the pond. A monk pushes the door in the moonlight.\u201d<\/em> Dao concentrated on the word \u201cpushes.\u201d He considered using \u201cknocks\u201d instead. After a long period of deliberation, he chose \u201cknocks\u201d because it provides contrast to the tranquil atmosphere of the suburban night. Such a careful re-thinking process led to a beautiful poem that continues to spread:<\/p>\n

\"\"<\/p>\n

The\u00a0story about Dao\u2019s deliberation over the words \u201cpushes\u201d and \u201cknocks\u201d demonstrates an important human cognitive process: To compose great literary works, the first draft is typically a base to build on and polish until it yields the final version. The same is true in the process of writing academic papers; first drafts are usually written by students and then revised or re-written by their supervisors. In these cases, the first draft is typically unsatisfactory but provides a global textual skeleton for retouching and revision. The polishing process, which we call deliberation, improves the quality.<\/p>\n

In a paper presented at the 2017 Neural Information Processing Systems (opens in new tab)<\/span><\/a> conference in Long Beach, California, we describe our Deliberation Network<\/a> that we developed to enhance artificial intelligence, in particular natural language generation. Our network is inspired by the human process of deliberation.<\/p>\n

We start with the basic neural machine translation structure: the sequence-to-sequence model, shown in Figure 1. To translate a sentence in a source language, for example the Chinese sentence \u201c\u5fae\u8f6f\u4e9a\u6d32\u7814\u7a76\u9662\u5373\u5c06\u8fce\u6765\u4e8c\u5341\u5468\u5e74\u8bde\u8fb0,\u201d we first use a neural network called an encoder to scan over it and then pass the output to another neural network called the decoder. Based on the encoder\u2019s output, the decoder will generate each translated word in the target language, for example, in this case, \u201cMicrosoft Research is about to have twenty birthday,\u201d which is a rough translation.<\/p>\n

\"\"

Figure 1: Basic sequence-to-sequence structure to perform translation.<\/em><\/p><\/div>\n

As Figure 1 shows, there is only one generation process in this framework. That is, the generation result via the decoder is set as the final translation. We refer to such a decoder as the first pass decoder. To add the deliberation process, we added another decoder, the second pass decoder, to the network structure, which is illustrated in Figure 2.<\/p>\n

\"\"

Figure 2: The structure of Deliberation Network.<\/em><\/p><\/div>\n

The second pass decoder expands the translation process to two steps: the first pass decoder first reads the encoder\u2019s encoding result for the source sentence x<\/em> and outputs a rough, or first-draft, translation sentence, denoted as y’<\/em>. Then, the second pass decoder takes both x<\/em> and y’<\/em> as inputs to generate the final translation y<\/em>. The second step is the deliberation process: the y’<\/em> output via the first pass decoder acts as the draft, which is revised into y<\/em> via automatic learning using a deep neural network. An example is shown in Figure 2: the original translation, \u201cMicrosoft Research is about to have twenty birthday\u201d is revised to: \u201cMicrosoft Research Asia will celebrate twentieth anniversary of birth.\u201d<\/p>\n

We verify the effectiveness of our Deliberation Network on a large benchmark dataset, the English-French translation campaign from the Ninth Workshop on Statistical Machine Translation (opens in new tab)<\/span><\/a> in 2014, which contains about 36 million training data and 3 thousand test data. The results are shown in Table 1. The baseline systems include several of the most powerful neural machine translation \u00a0systems in the current literature such as We conclude that, together with our previous dual learning technique to effectively leverage monolingual data, we achieve the best single model performance (41.50) on this task using Deliberation Network, on top of a simple stack LSTM architecture (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n\n\n\n\n\n\n
Table 1: Comparison between Deliberation Network and different deep NMT systems on WMT’14 En->Fr.<\/em><\/caption>\n
System<\/strong><\/td>\nConfigurations<\/strong><\/td>\nBLEU<\/strong><\/td>\n<\/tr>\n
GNMT (opens in new tab)<\/span><\/a><\/td>\n8-8 stacked LSTM encoder and decoder + RL finetune<\/td>\n39.92<\/td>\n<\/tr>\n
FairS2S (opens in new tab)<\/span><\/a><\/td>\n15-15 convolutional encoder and decoder<\/td>\n40.51<\/td>\n<\/tr>\n
Transformer (opens in new tab)<\/span><\/a><\/td>\n6-6 self-attention encoder and decoder<\/td>\n41.0<\/td>\n<\/tr>\n
This work<\/td>\n4-4 stacked LSTM encoder and decoder<\/td>\n39.51<\/td>\n<\/tr>\n
+Dual learning<\/a><\/td>\n40.53<\/td>\n<\/tr>\n
+Dual learning + Deliberation Network<\/td>\n41.5<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Related<\/strong>:<\/p>\n