What's Wrong With This Translation?: Simplifying Error Annotation For Crowd Evaluation

Iben Nyholm Debess; Alina Karakanta; Barbara Scalvini

What's Wrong With This Translation? Simplifying Error Annotation For Crowd Evaluation

Iben Nyholm Debess, Alina Karakanta, Barbara Scalvini

Research output: Contribution to conference › Paper › peer-review

Abstract

Machine translation (MT) for Faroese faces challenges due to limited expert annotators and a lack of robust evaluation metrics. This study addresses these challenges by developing an MQM-inspired expert annotation framework to identify key error types and a simplified crowd evaluation scheme to enable broader participation. Our findings based on an analysis of 200 sentences translated by three models demonstrate that simplified crowd evaluations align with expert assessments, paving the way for improved accessibility and democratization of MT evaluation.

Original language	English
Pages	42-47
Number of pages	6
Publication status	Published - 2025
Event	NB-REAL – Nordic-Baltic Responsible Evaluation and Alignment of Language models - Tallinn, Estonia Duration: 2 Mar 2025 → 2 Mar 2025 https://nbreal.xyz/

Workshop

Workshop	NB-REAL – Nordic-Baltic Responsible Evaluation and Alignment of Language models
Abbreviated title	NB-REAL
Country/Territory	Estonia
City	Tallinn
Period	2/03/25 → 2/03/25
Internet address	https://nbreal.xyz/

Keywords

Machine translation

Access to Document

https://hdl.handle.net/10062/107160

Cite this

@conference{2cdd68bce46f49968aa264c8be768735,

title = "What's Wrong With This Translation?: Simplifying Error Annotation For Crowd Evaluation",

abstract = "Machine translation (MT) for Faroese faces challenges due to limited expert annotators and a lack of robust evaluation metrics. This study addresses these challenges by developing an MQM-inspired expert annotation framework to identify key error types and a simplified crowd evaluation scheme to enable broader participation. Our findings based on an analysis of 200 sentences translated by three models demonstrate that simplified crowd evaluations align with expert assessments, paving the way for improved accessibility and democratization of MT evaluation.",

keywords = "Machine translation",

author = "Debess, {Iben Nyholm} and Alina Karakanta and Barbara Scalvini",

year = "2025",

language = "English",

pages = "42--47",

note = "NB-REAL – Nordic-Baltic Responsible Evaluation and Alignment of Language models, NB-REAL ; Conference date: 02-03-2025 Through 02-03-2025",

url = "https://nbreal.xyz/",

}

TY - CONF

T1 - What's Wrong With This Translation?

T2 - NB-REAL – Nordic-Baltic Responsible Evaluation and Alignment of Language models

AU - Debess, Iben Nyholm

AU - Karakanta, Alina

AU - Scalvini, Barbara

PY - 2025

Y1 - 2025

N2 - Machine translation (MT) for Faroese faces challenges due to limited expert annotators and a lack of robust evaluation metrics. This study addresses these challenges by developing an MQM-inspired expert annotation framework to identify key error types and a simplified crowd evaluation scheme to enable broader participation. Our findings based on an analysis of 200 sentences translated by three models demonstrate that simplified crowd evaluations align with expert assessments, paving the way for improved accessibility and democratization of MT evaluation.

AB - Machine translation (MT) for Faroese faces challenges due to limited expert annotators and a lack of robust evaluation metrics. This study addresses these challenges by developing an MQM-inspired expert annotation framework to identify key error types and a simplified crowd evaluation scheme to enable broader participation. Our findings based on an analysis of 200 sentences translated by three models demonstrate that simplified crowd evaluations align with expert assessments, paving the way for improved accessibility and democratization of MT evaluation.

KW - Machine translation

M3 - Paper

SP - 42

EP - 47

Y2 - 2 March 2025 through 2 March 2025