Human-LLM Alignment in Language Attitudes Toward Non-Native Japanese
Naho Orita, Hayato Ogawa, Daisuke Kawahara
Why It Matters
What makes this one worth your time
Understanding how LLMs replicate or diverge from human biases in language evaluation is crucial for fair and equitable AI applications in high-stakes domains.
The study reveals that LLMs mimic human biases in evaluating non-native Japanese writing but differ in certain social dimensions.
Summary
The paper investigates how large language models (LLMs) and human raters evaluate non-native Japanese writing, focusing on fluency, status, and solidarity. It finds that LLMs replicate human biases but with less emphasis on solidarity and more differentiation based on the learner's native language.
Key contributions
- Comparison of human and LLM evaluations of non-native Japanese writing.
- Identification of bias patterns in LLMs that mirror human attitudes.
- Application of the language attitudes framework to audit LLMs beyond English.
Notable insights
- LLMs understate the solidarity gap, indicating a potential area for improving social sensitivity in AI evaluations.
- LLMs differentiate among learner L1 backgrounds, unlike human raters, suggesting a nuanced bias that may need addressing.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2608.01629v1 Announce Type: new Abstract: Large language models (LLMs) increasingly evaluate human writing in high-stakes domains such as hiring and academic assessment, putting non-native speakers at particular risk. Drawing on the language attitudes framework, we compared human and LLM evaluations of parallel L1- and L2-written Japanese emails on three dimensions: fluency, status, and solidarity. Japanese raters rated L2 texts significantly lower on all three dimensions, with a fluency gap roughly twice the size of the status and solidarity gaps. Six LLM judges reproduced the direction of this bias, and five reproduced its ordering across dimensions. The models diverged from humans in two ways: all understated the solidarity gap, the most socially grounded dimension, and all differentiated among learner L1 backgrounds where humans did not. LLM judges thus reproduce native speakers' language attitudes in a structured yet attenuated form, and the language attitudes framework offers a ready-made yardstick for auditing them beyond English.