Lemmatization of low-resource languages in diachronic linguistics: Problems and solutions

Abstract

Introduction. The article addresses the problem of lemmatizing low‑resource historical languages within applied diachronic linguistics. Standard approaches widely used for modern texts — rule‑based models and neural architectures — are inapplicable for ancient languages because of their morpho- logical complexity and the scarcity of corpus data. As a result, dictionary‑based lemmatization remains the most effective strategy. For Old English, the widely available Classical Language Toolkit (CLTK) offers only limited functionality, since its lemma dictionary provides insufficient coverage. The aim of this study is to compile an expanded lemma dictionary of Old English using data from the crowdsourced lexicographic resource Wiktionary, thereby enabling more accurate automatic lemmatization.
Materials and Methods. The study draws on the annotated treebank corpus of Old English texts (9th–11th centuries) and the datasets from the open lexicographic resources (the CLTK lemma list and the Old English segment of Wiktionary). The methodological framework combines methods of corpus and computational linguistics, and electronic lexicography. The research consists of three stages: compilation of an Old English lemma dictionary, lemmatization of Old English texts, and evaluation of the dictionary’s precision and recall.
Results. The analysis of the lemmatizers for low‑resource languages demonstrates that the dictionary‑based approach proves to be more effective. Traditional lexicographic resources such as Bosworth– Toller remain valuable, but their lack of machinereadable formats limits direct application. As an alternative we propose Wiktionary — a free, crowdsourced lexicographic resource characterized by both broad lexical coverage and detailed descriptions of lemmas. Within the framework of this study, we developed a compiled lemma dictionary by integrating CLTK lemma list and Wiktionary datasets. The resulting lemma dictionary (11,451 unique lemmas, 80,778 wordforms) demonstrates a high degree of precision and recall, confirming its applicability for the lemmatization of Old English texts.
Conclusion. The study thus provides a new digital resource — an Old English lemma dictionary accompanied by Python code for fully automatic lemmatization. Together, these tools represent an effective solution to the lemmatization of a low-resource historical language and contribute to the development of computational diachronic linguistics.