This work is concerned with large language models (LLMs), the current dominant paradigm in natural language processing in research and industrial settings, thanks to those models’ generalization capabilities. Although LLMs have generated wide excitement, many obstacles are left before they can be deployed everywhere, not the least of which is the fact that state-of-the-art results are, for now, only possible in English, with other languages far behind. We try to make progress towards better models in languages with lower amounts of linguistic resources. In the first half, we create artifacts: we process and crowdsource a large multilingual dataset for language modeling and document the tools required for this, and then we train the first open massive multilingual language model as part of a large international research collaboration. In the latter half, we propose empirically derived recommendations for practitioners in low-resource pretraining and finetuning settings. We estimate the performance degradation that results from training on multiple epochs in compute-rich, data-constrained settings and find that it is broadly acceptable. We then show that the prompt-following capabilities of language models can be combined with the pretrain-then-finetune paradigm in NLP and that this helps particularly in settings where downstream task data is scarce.

Jury members

• Stephan Oepen, Head of the Department for Informatics, University of Oslo (Reviewer)
• Jörg Tiedemann, Professor of Language Technology at the Department of Digital Humanities Languages, University of Helsinki (Reviewer)
• François Yvon, Senior researcher (HDR), CNRS, Université Paris-Sorbonne
• Colin Raffel, Assistant Professor in the Department of Computer Science University of North Carolina, Chapel Hill
• Sasha Rush, Associate Professor, Cornell University (PhD advisor)
• Claire Gardent, CNRS Director of Research, LORIA (PhD advisor)