Translation quality aspects are heterogeneous and diverse, involving, in general, many different linguistic dimensions. However, most automatic evaluation methods in use today rely on partial quality assumptions, such as lexical similarity. This introduces a bias in the development cycle which in some cases has been reported to carry very negative consequences. In order to tackle this methodological problem, we explore a novel path towards heterogeneous automatic MT evaluation. We have compiled a rich set of specialized similarity metrics operating at different linguistic levels (lexical, syntactic and semantic). We have also studied how the scores conferred by different metrics may be integrated into a single measure of quality, without having to adjust their relative importance.

This demo allows you to obtain automatic evaluation scores according to a selected set of metric representatives, together with ULC combined score (i.e., arithmetic mean) over a heuristically defined set of metrics.

• Select the target language. The metric set will depend on this choice. Currently, linguistic features are only supported for English. For other languages, the metric set limits to the lexical dimension.
• Type test cases in the text areas below:
• at least one, and up to five, candidate translations.
• at least one, and up to five, reference translations.
• text areas must contain one test case per line up to a maximum of ten.
• Click on the "Go Evaluate!" button at the right.
• Execution should take between one and five minutes for English, and a just a few seconds for other languages.