Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation
Lester James V. Miranda, Ivan Vuli\'c, Anna Korhonen
Why It Matters
What makes this one worth your time
Understanding which language models serve as effective teachers can enhance the quality of synthetic data generation, particularly for less-resourced languages, thereby improving multilingual model performance.
This research identifies effective language models for generating multilingual synthetic data and offers practical recommendations for improving model training.
Summary
The paper evaluates the effectiveness of various language models as teachers for generating multilingual synthetic data, introducing the Polyglot Score metric and analyzing data quality and student model performance across multiple languages.
Key contributions
- Introduction of the Polyglot Score for measuring data quality and student performance.
- Systematic evaluation of 10 language models across 6 languages to identify effective teachers.
- Practical recommendations for matching teacher-student model pairs and improving data generation techniques.
Notable insights
- Model scale does not predict teacher effectiveness; data qualities like prompt diversity and response fluency are more significant.
- The study generates over 1.4M examples and trains 240 student models, providing a robust evaluation framework.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2604.11290v2 Announce Type: replace Abstract: Synthesizing supervised finetuning (SFT) data from language models (LMs) to teach smaller models multilingual tasks has become increasingly common. However, teacher model selection is often ad hoc, typically defaulting to the largest available option, even though such models may have significant capability gaps in non-English languages. This practice can result in poor-quality synthetic data and suboptimal student downstream performance. In this work, we systematically characterize what makes an effective multilingual teacher. We measure intrinsic measures of data quality with extrinsic student model performance in a metric we call Polyglot Score; evaluating 10 LMs across 6 typologically diverse languages, generating over 1.4M SFT examples and training 240 student models. Among the models tested, Gemma 3 27B and Aya Expanse 32B emerge as consistently effective teachers across different student base model families. Further analyses reveal that model scale alone does not significantly predict teacher effectiveness; instead, data qualities such as prompt diversity, length, and response fluency capture over 93.3% of variance in intrinsic data quality and predict student performance. Finally, we provide practical recommendations, including matching the model families of teacher-student pairs and translating from or responding to existing prompts, which can yield improvements for less-resourced languages. We hope that our work advances data-centric research in multilingual synthetic data and LM development.