ParaEval: Using Paraphrases to Improve Machine Translation and Summarization Evaluations

April 20, 2006
Liang Zhou | Information Sciences Institute (ISI)

The machine translation and text summarization communities have both benefitted greatly from the introduction of automated evaluation procedures. Automatic evaluation methods facilitate faster turnaround for system development and assessment cycles, thus making them an essential part of the research problem. However, at the present, methodologies used in the evaluations are limited to lexical identity matching. The lack of support for word or phrase matching that stretches beyond strict lexical matches has limited the expressiveness and utility of these methods. In this talk, I present ParaEval, an automatic evaluation framework that uses paraphrases to improve machine translation and summarization evaluations. A large collection of paraphrases is collected through an unsupervised acquisition process, inspired by phrase-based statistical machine translation. I will focus on ParaEval’s support for paraphrase/synonym matching, recall measurement, and correlation with human judgments.

Speaker Details

Liang Zhou is a graduating Ph.D. student from the Information Sciences Institute (ISI). During her doctoral study, she has worked on a breadth of natural language processing and computational linguistics topics, including summarization (text and conversation), machine translation, and evaluations. She has also worked on speech processing (transcripts from automatic speech recognition) during an internship with the University of Toronto’s Computational Linguistics Group.