We’re excited to announce that, as of today, masculine and feminine alternative translations are available for when translating from English to Spanish, French, or Italian. You can try out this new feature in both Bing Search and Bing Translator verticals.

Over the last few years, the field of Machine Translation (MT) has been revolutionized by the advent of transformer models, leading to tremendous improvements in quality. However, models optimized to capture the statistical properties of data collected from the real world inadvertently learn or even amplify social biases found in that data.

Our latest release is a step towards reducing one of these biases, specifically gender bias that is prevalent in MT systems. Bing Translator has always produced a single translation for an input sentence even when the translations could have had other gender variations including feminine and masculine variants. In accordance with the Microsoft responsible AI principles, we want to ensure we provide correct alternative translations and are more inclusive to all genders. As part of this journey our first step is to provide feminine and masculine translation variants.

Gender is expressed differently across different languages. For example, in English, the word lawyer could refer to either a male or female individual, but in Spanish, abogada would refer to a female lawyer, while abogado would refer to a male one. In the absence of information about the gender of a noun like ‘lawyer’ in a source sentence, MT models may resort to selecting an arbitrary gender for the noun in the target language. Often, these arbitrary gender assignments align with stereotypes, perpetuating harmful societal bias (Stanovsky et al., 2019; Ciora et al., 2021) and leading to translations that are not fully accurate.

In the example below, you notice that while translating gender-neutral sentences from English to Spanish, the translated text follows the stereotypical gender role, i.e., lawyer is translated as being male.

As there is no context in the source sentence that implies the gender of the lawyer, producing a translation with the assumption of either a male or female lawyer would both be valid. Now, Bing Translator produces translations with both feminine and masculine forms.

System design

We aimed to design our system to meet the following key criteria for providing gendered alternatives:

1. The feminine and masculine variants should have minimal differences except for those needed to convey gender.
2. We wanted to cover a wide range of sentences where multiple gendered alternatives are possible.
3. We wanted to ensure that the translations preserve the meaning of the original source sentence.

Detecting gender ambiguity

In order to accurately detect gender ambiguity in source text, we utilize a coreference model to analyze inputs containing animate nouns. For instance, if a given input text contains a gender-neutral profession word, we only want provide gendered alternatives for it when its gender can’t be determined by other information in the sentence. For example: On translating an English sentence “The lawyer met her driver at the hotel lobby.” into French we can determine that the lawyer is female, while the gender of the driver is unknown.

Generating alternate translation

When the source sentence is ambiguously gendered, we examine our translation system’s output to decide if an alternative gender interpretation is possible. If so, we proceed to determine the best way to revise the translation. We begin by constructing a set of candidate target translations by rewriting the original translation. We apply linguistic constraints based on dependency relations to ensure consistency in the proposed alternatives and prune the erroneous candidates.

However, in many cases, even after applying our constraints, we are left with multiple candidate rewrites for the gendered alternative translation. To determine the best option, we evaluate each candidate by scoring it with our translation model. By leveraging the fact that a good gender rewrite will also be an accurate translation of the source sentence, we are able to ensure high accuracy in our final output.

Leveraging managed online endpoints in Azure Machine Learning

The gendered alternative feature in Bing is hosted on managed online endpoints in Azure Machine Learning. Managed online endpoints provide a unified interface to invoke and manage model deployments on Microsoft-managed compute in a turnkey manner. They enable us to take advantage of scalable and reliable endpoints without being concerned about infrastructure management. This inference environment also enables the processing of large numbers of requests with low latency. Our ability to create and deploy the gender debias service with the latest frameworks and technologies has been greatly improved through the use of managed inference features in Azure Machine Learning. By leveraging these features, we have been able to maintain low COGS (Cost of Goods Sold) and ensure straightforward security and privacy compliance.

How can you contribute?

To facilitate progress in gender bias reduction in MT, we are releasing a test corpus containing gender-ambiguous translation examples from English into Spanish, French and Italian. Each English source sentence is accompanied by multiple translations, covering each possible gender variation.

Our test set is constructed to be challenging, morphologically rich and linguistically diverse. This corpus has been instrumental in our development process. It was developed with the help of a bilingual linguists with significant translation experience. We are also releasing a technical paper that discusses the test corpus in detail and the methodology and tools for evaluation.

GATE: A challenge set for Gender-Ambiguous Translation Examples – Paper

GATE: A challenge set for Gender-Ambiguous Translation Examples – Test set

Path forward

Through this work we aim to improve the quality of MT output in cases of ambiguous source gender, as well as facilitate the development of better and more inclusive natural language processing (NLP) tools in general. Our initial release focuses on translating from English to Spanish, French, and Italian. Going forward, we plan to expand to new language pairs, as well as cover additional scenarios and types of biases.

Credits:

Ranjita Naik, Spencer Rarrick, Sundar Poudel, Varun Mathur, Jeshwanth Kumar Chandrala, Charan Mohan, Lee Schwartz, Steven Nguyen, Amit Bhagwat, Vishal Chowdhary.