Prompt Engineering Enhances Faroese MT, but Only Humans Can Tell

Barbara Scalvini; Annika Simonsen; Iben Nyholm Debess; Hafsteinn Einarsson

Prompt Engineering Enhances Faroese MT, but Only Humans Can Tell

Barbara Scalvini, Annika Simonsen, Iben Nyholm Debess, Hafsteinn Einarsson

Research output: Contribution to conference › Paper › peer-review

Abstract

This study evaluates GPT-4's English-to-Faroese translation capabilities, comparing it with multilingual models on FLORES-200 and Sprotin datasets. We propose a prompt optimization strategy using Semantic Textual Similarity (STS) to improve translation quality. Human evaluation confirms the effectiveness of STS-based few-shot example selection, though automated metrics fail to capture these improvements. Our findings advance LLM applications for low-resource language translation while highlighting the need for better evaluation methods in this context.

Original language	English
Pages	622-633
Number of pages	12
Publication status	Published - 2025
Event	Nodalida/Baltic-HLT 2025: The Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies - Tallinn, Estonia Duration: 2 Mar 2025 → 5 Mar 2025 Conference number: 25 https://www.nodalida-bhlt2025.eu/conference

Conference

Conference	Nodalida/Baltic-HLT 2025
Country/Territory	Estonia
City	Tallinn
Period	2/03/25 → 5/03/25
Internet address	https://www.nodalida-bhlt2025.eu/conference

Keywords

Machine learning
translations
GPT-4

Access to Document

Cite this

@conference{69d5b491ae964a92919d88507a5a3512,

title = "Prompt Engineering Enhances Faroese MT, but Only Humans Can Tell",

abstract = "This study evaluates GPT-4's English-to-Faroese translation capabilities, comparing it with multilingual models on FLORES-200 and Sprotin datasets. We propose a prompt optimization strategy using Semantic Textual Similarity (STS) to improve translation quality. Human evaluation confirms the effectiveness of STS-based few-shot example selection, though automated metrics fail to capture these improvements. Our findings advance LLM applications for low-resource language translation while highlighting the need for better evaluation methods in this context.",

keywords = "Machine learning, translations, GPT-4",

author = "Barbara Scalvini and Annika Simonsen and Debess, \{Iben Nyholm\} and Hafsteinn Einarsson",

year = "2025",

language = "English",

pages = "622--633",

note = "Nodalida/Baltic-HLT 2025 : The Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies ; Conference date: 02-03-2025 Through 05-03-2025",

url = "https://www.nodalida-bhlt2025.eu/conference",

}

TY - CONF

T1 - Prompt Engineering Enhances Faroese MT, but Only Humans Can Tell

AU - Scalvini, Barbara

AU - Simonsen, Annika

AU - Debess, Iben Nyholm

AU - Einarsson, Hafsteinn

N1 - Conference code: 25

PY - 2025

Y1 - 2025

N2 - This study evaluates GPT-4's English-to-Faroese translation capabilities, comparing it with multilingual models on FLORES-200 and Sprotin datasets. We propose a prompt optimization strategy using Semantic Textual Similarity (STS) to improve translation quality. Human evaluation confirms the effectiveness of STS-based few-shot example selection, though automated metrics fail to capture these improvements. Our findings advance LLM applications for low-resource language translation while highlighting the need for better evaluation methods in this context.

AB - This study evaluates GPT-4's English-to-Faroese translation capabilities, comparing it with multilingual models on FLORES-200 and Sprotin datasets. We propose a prompt optimization strategy using Semantic Textual Similarity (STS) to improve translation quality. Human evaluation confirms the effectiveness of STS-based few-shot example selection, though automated metrics fail to capture these improvements. Our findings advance LLM applications for low-resource language translation while highlighting the need for better evaluation methods in this context.

KW - Machine learning

KW - translations

KW - GPT-4

M3 - Paper

SP - 622

EP - 633

T2 - Nodalida/Baltic-HLT 2025

Y2 - 2 March 2025 through 5 March 2025

ER -

Prompt Engineering Enhances Faroese MT, but Only Humans Can Tell

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this