ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation
Ekaterina Trofimova, Zosia Shamina, Maria Selifanova, Artem Zaitsev, Remi Savchuk, Maxim Minets, Daria Ozerova, Emil Sataev, Denis Zuenko, Andrey E. Ustyuzhanin
Why It Matters
What makes this one worth your time
Understanding how language models handle cross-lingual tasks in ML pipeline generation is crucial for global AI deployment and ensuring equitable performance across languages.
ML2B benchmark evaluates LLMs' cross-lingual ML pipeline generation capabilities.
Summary
The paper introduces ML2B, a benchmark designed to evaluate the ability of large language models to generate machine learning pipelines from task descriptions in multiple languages. It includes 35 Kaggle competitions translated into 14 languages, resulting in 490 task-language pairs, and uses a network-isolated evaluation infrastructure. The study reveals that cross-lingual performance varies significantly depending on the task, challenging assumptions about multilingual model capabilities.
Key contributions
- Introduction of the ML2B benchmark for cross-lingual ML pipeline generation.
- Evaluation of LLMs on 490 task-language pairs from 35 Kaggle competitions.
- Provision of standardized evaluation protocols and an AutoGluon baseline.
Notable insights
- Cross-lingual performance degradation is highly task-dependent rather than following traditional resource-availability hierarchies.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2509.22768v3 Announce Type: replace Abstract: We introduce ML2B, the first benchmark for evaluating cross-lingual task comprehension in end-to-end ML pipeline generation by large language models. Despite growing global AI adoption, no systematic evaluation exists for ML pipeline generation beyond English task descriptions. ML2B addresses this gap with 35 Kaggle competitions spanning tabular, text, and image domains, translated into 14 languages by native-speaker researchers with ML expertise, yielding 490 task-language pairs. To ensure evaluation integrity, the benchmark incorporates 10 private competitions without publicly available solutions and employs network-isolated evaluation infrastructure restricting runtime access to essential ML resources. We provide standardized evaluation protocols, an AutoGluon algorithmic baseline, and comprehensive failure mode analysis. Experiments with frontier models (GPT-4.1-mini, GPT-OSS-120b, Gemini-2.5-Flash) reveal that cross-lingual performance degradation is highly task-dependent rather than following traditional resource-availability hierarchies, with gaps ranging from language advantages to severe degradation depending on competition characteristics. These findings challenge conventional assumptions about multilingual model capabilities and underscore the necessity of systematic cross-lingual evaluation for ML pipeline generation. We open-source the benchmark, baselines, and evaluation infrastructure at https://github.com/enaix/ml2b.