ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs
Andrianos Michail, Stylianos Psychias, Michelle Wastl, Simon Clematide, Rico Sennrich, Juri Opitz
Why It Matters
What makes this one worth your time
Understanding and improving cross-lingual semantic representation is crucial for developing more inclusive and effective language models, especially for low-resource languages.
ALEE offers a new framework for evaluating text embeddings in any language using English-centric minimal pairs.
Summary
The paper introduces ALEE, a framework for evaluating text embeddings across multiple languages using English-centric minimal pairs and Abstract Meaning Representations. It aims to address limitations in current benchmarks by providing a more dynamic, language-inclusive evaluation tool that highlights cross-lingual semantic representation gaps.
Key contributions
- Introduction of ALEE, a framework for cross-lingual embedding evaluation.
- Application of Abstract Meaning Representations to create English minimal pairs.
- Empirical study across 275+ languages and three parallel datasets.
Notable insights
- Using Abstract Meaning Representations to generate minimal pairs allows for controlled semantic shifts in evaluation.
- The framework highlights performance disparities linked to language prevalence in training resources and subword tokenization.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.00171v1 Announce Type: new Abstract: Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover only a limited set of languages, are often domain-specific, susceptible to overfitting, and poorly representative of low-resource languages. To address these limitations, we introduce ALEE, a framework that extends Sentence Smith (Li et al., 2025) to the cross-lingual and paragraph level. ALEE uses Abstract Meaning Representations (AMR) to generate English minimal pairs with controlled, fine-grained semantic shifts, which are paired with translations in target languages. This approach enables targeted diagnostics for models in any language with English parallel data. We conduct a large-scale empirical study across a diverse set of embedding models and 275+ languages spanning three parallel datasets. On ALEE, performance varies substantially across languages, text lengths, and linguistic phenomena, exposing persistent gaps in cross-lingual semantic representation that track language prevalence in training resources and subword tokenization. We release ALEE at https://github.com/Andrian0s/any-lang-embed-eval