Expectation Alignment of Language Models for Real-World User Expectations
Miaomiao Li, Yang Wang, Bin Liang, Shudong Liu, Zhiwei Zhang, Kam-Fai Wong
Why It Matters
What makes this one worth your time
Understanding and aligning with real user expectations is crucial for improving the practical utility and user satisfaction of language models.
The paper proposes a new benchmark and framework to better align language models with user expectations.
Summary
The paper investigates the alignment of large language models with real-world user expectations, introducing a benchmark called ExpectBench and a framework named LENS to improve model responses by internalizing user expectations.
Key contributions
- Introduction of ExpectBench, a benchmark based on real user expectations.
- Proposal of LENS, a framework for expectation-aware response generation.
Notable insights
- The introduction of ExpectBench highlights the gap between model performance on standard benchmarks and real user expectations.
- LENS framework suggests a novel approach to incorporate latent user expectations into response generation.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.20485v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated remarkable performance on standard benchmarks, yet it remains largely unexplored whether they truly meet user expectations. Existing evaluation approaches, relying on model heuristics, expert rubrics, or user simulation, fail to capture the diversity and subtlety of real human expectations, causing models to appear competent while misaligning with what users actually seek. We present the first systematic study of user expectations in real-world LLM interactions, proposing a principled procedure to extract semantically rich expectations and introducing ExpectBench, a benchmark grounded in real user expectations. Analyses reveal that current LLMs struggle to satisfy and anticipate what users hope to obtain, highlighting a fundamental source of misalignment. Building on these observations, we propose LENS, a lightweight latent expectation-aware response generation framework. LENS enables models to internalize user expectations and generate better-aligned responses, consistently improving expectation satisfaction and underscoring the importance of explicitly modeling user expectations for realistic human-AI alignment.