Evaluating the potential of language-family-specific generative models for low resource data augmentation: a Faroese case study

We investigate GPT-Sw3, a generative language model for the Nordic languages, to assess its understanding of low-resource Faroese. Our aim is to demonstrate the advantages of using language-family-specific generative models to augment data for related languages with fewer resources. We evaluate GPT-Sw3 by prompting it for Faroese to English translation in a zero, one and few-shot setting. We assess such translations with an ensemble score consisting of an arithmetic average between the BLEU and a semantic similarity score (SBERT). Moreover, we challenge the model’s Faroese language understanding capabilities on a small dataset of curated Faroese trick sentences. There, we compare the model’s performance with Open AI’s GPT 3.5 and GPT 4, demonstrating the advantages of using a language family specific generative model for navigating non trivial scenarios. We evaluate the pipeline thus created and use it, as a proof of concept, to create an automatically annotated Faroese semantic textual similarity (STS) dataset.