Comparing Large Language Models on Scrum Certification-Style Questions: Accuracy, Stability, and Error Patterns
Robson Alves Vilar, Emanuel Dantas Filho, Ademar Fran\c{c}a de Sousa Neto, Mirko Perkusich, Danyllo Wagner Albuquerque, Jo\~ao Paiva, Kyller Gorg\^onio, Angelo Perkusich
Why It Matters
What makes this one worth your time
Understanding how LLMs perform on domain-specific certification questions can guide improvements in AI-driven educational tools and assessments.
The paper compares LLMs on Scrum certification questions, highlighting accuracy and error patterns.
Summary
The paper evaluates the performance of three large language models on Scrum certification-style questions, analyzing their accuracy, stability, and error patterns across different prompting strategies and question formats.
Key contributions
- Evaluation of LLMs on a specific domain of Scrum certification questions.
- Analysis of error patterns and performance across different question formats and topics.
Notable insights
- Models showed systematic errors related to overgeneralization and conflicts with strict Scrum definitions.
- Gemini 3 Flash outperformed other models, indicating potential model-specific strengths in handling structured knowledge.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.00048v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used in exam- and certification-style question answering tasks, where their ability to retrieve, interpret, and apply domain-specific knowledge can be systematically assessed. In Software Engineering, such settings are particularly relevant when questions depend on strict adherence to normative definitions, roles, artifacts, and rules. This paper evaluates the performance of three contemporary LLMs, \textit{GPT-5 mini}, \textit{Gemini 3 Flash}, and \textit{DeepSeek Chat 3.2}, in answering 993 Scrum certification-style questions aligned with the Professional Scrum Master I (PSM I) assessment format. We evaluated the models under three prompting strategies (\textit{zero-shot}, \textit{chain-of-thought}, and \textit{source-grounded}), with repeated executions to assess intra-model stability. We also analyzed performance across Scrum topics and question formats, complemented by a qualitative analysis of recurring error patterns in incorrect answers. Results revealed clear differences among models, with Gemini 3 Flash achieving the highest accuracy, followed by GPT-5 mini and DeepSeek Chat 3.2, while intra-model variability remained low across all conditions. By question format, the models achieved the highest accuracy on single-answer multiple-choice items, whereas multi-select and True/False questions were more error-prone. By topic, performance was more consistent in normatively explicit areas such as Artifacts, Empiricism, and Product Value, but more fragile in Scrum Values, Self-Managing Teams, and Stakeholders \& Customers. The qualitative analysis showed that errors were systematic rather than random, involving overgeneralization, restrictive wording, compound distractors, and conflicts between common market interpretations and strict Scrum definitions.