Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth
Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmad ElShiekh, Clayton W. Taylor, Ahmed Rashad
Why It Matters
What makes this one worth your time
Understanding and improving LLM performance in culturally nuanced and linguistically diverse contexts is crucial for deploying these models in real-world applications involving underrepresented languages.
A framework for evaluating Arabic dialect knowledge in LLMs reveals challenges in cultural task grading.
Summary
The paper presents a cross-evaluation framework for assessing Arabic cultural and sociolinguistic knowledge in language models, focusing on Egyptian and Iraqi dialects. It introduces 103 prompt-rubric pairs graded by native speakers and evaluates three frontier LLMs using human SMEs and five automated judges. The study highlights challenges in grading cultural tasks and identifies implicit cultural reasoning as a key failure mode.
Key contributions
- Development of a cross-evaluation framework for Arabic dialects.
- Introduction of 103 validated prompt-rubric pairs for Egyptian and Iraqi Arabic.
- Evaluation of LLMs using a dual-metric scheme to separate grading bias from noise.
Notable insights
- Implicit cultural reasoning is a primary failure mode for automated grading in LLMs.
- Cultural tasks are consistently harder to grade than linguistic tasks across different judges.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.00139v1 Announce Type: new Abstract: The cost of human expert evaluation is a principal bottleneck to deploying language models in specialized, high-stakes domains. This is particularly acute for Arabic sociolinguistic knowledge: credible grading requires not only linguistic fluency but deep cultural familiarity that cannot be approximated by surface-level metrics. We address this with a cross-evaluation framework instantiated on two underrepresented Arabic dialect communities: Egyptian and Iraqi Arabic. We contribute 103 validated prompt-rubric pairs (70 Egyptian, 33 Iraqi; 53 Cultural, 50 Linguistic), authored and graded by native-speaker SMEs using penalty-weighted rubrics distinguishing positive content requirements from answer-specific negative error criteria. Three frontier LLMs serve as target models (graded by human SMEs across 302 unique prompt-response pairs), while five frontier LLMs serve as automated judges enforcing a provider-level self-evaluation guard. A dual-metric scheme combining Mean Absolute Deviation (MAD) with Signed Mean Error separates directional grading bias from symmetric noise. Across 1,307 judge evaluations: GPT-5.4 is the most reliable judge (MADj = 10.21 pp, Signed Error = -1.12%); four of five judges show systematic leniency (+2.01% to +6.56%); Cultural tasks are harder to grade than Linguistic tasks for all judges (MAD gap 1.83-4.78 pp); and models substantially outperform on Egyptian prompts compared to Iraqi prompts. However, given leniency differences between Iraqi and Egyptian SMEs, we cannot solely attribute this gap to model knowledge. We therefore emphasize findings that do not assume identical leniency across human graders. Across all samples, implicit cultural reasoning -- requiring models to simulate native-speaker judgment rather than rely on lexical verification -- emerges as the primary failure mode for automated grading across all judge models.