SWE-IF: Aligning Code Evaluation with Human Preference
Ming Zhong, Xiang Zhou, Ting-Yun Chang, Qingze Wang, Nan Xu, Xiance Si, Dan Garrette, Shyam Upadhyay, Jeremiah Liu, Jiawei Han, Benoit Schillings, Jiao Sun
Why It Matters
What makes this one worth your time
Understanding and improving LLMs' ability to follow code instructions can lead to more user-aligned and effective coding assistance tools.
SWE-IF evaluates LLMs on instruction following and functional correctness to better align with human coding preferences.
Summary
The paper introduces SWE-IF, a testbed designed to evaluate large language models (LLMs) on both functional correctness and their ability to follow code instructions, which are hypothesized to align better with human preferences. The authors present VeriCode, a taxonomy of 30 verifiable code instructions, and use it to assess 31 LLMs, finding that instruction following is a key differentiator among models.
Key contributions
- Introduction of SWE-IF, a testbed for evaluating LLMs on instruction following and functional correctness.
- Development of VeriCode, a taxonomy of 30 verifiable code instructions with deterministic verifiers.
- Evaluation of 31 LLMs, highlighting their struggles with instruction compliance and functional regression.
Notable insights
- Instruction following is identified as a primary differentiator among LLMs, beyond functional correctness.
- A composite score of functional correctness and instruction following correlates best with human preference.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2510.07315v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check. Vibe check reflects human preference and goes beyond functionality: the solution should feel right, read cleanly, preserve intent, and remain correct. However, current code evaluation remains anchored to pass@k and captures only functional correctness, overlooking non-functional instructions that users routinely apply. In this paper, we hypothesize that instruction following is the missing piece underlying vibe check besides functional correctness. To quantify models' code instruction-following capabilities with measurable signals, we present VeriCode, a taxonomy of 30 verifiable code instructions together with deterministic verifiers. We use the taxonomy to augment established evaluation suites, resulting in SWE-IF, a testbed to assess both instruction following and functional correctness. Evaluating 31 LLMs, we show that even the strongest models struggle to comply with multiple instructions and exhibit functional regression. Most importantly, a composite score of functional correctness and instruction following correlates best with human preference, with instruction following emerging as the primary differentiator among LLMs. Our code, data, and taxonomy are available at https://github.com/maszhongming/SWE-IF.