Automated Creativity Evaluation of Language Models Across Open-Ended Tasks
Min Sen Tan, Zachary Kit Chun Choy, Syed Ali Redha Alsagoff, Nadya Yuki Wangsajaya, Mohor Banerjee, Swaagat Bikash Saikia, Alvin Chan
Why It Matters
What makes this one worth your time
This work addresses the need for scalable and generalizable metrics in assessing the creative capabilities of language models, which is crucial for advancing creative AI applications.
A novel framework for automated creativity evaluation of LLMs across diverse tasks.
Summary
The paper presents an automated, domain-agnostic framework for evaluating the creativity of large language models across open-ended tasks, utilizing metrics like semantic entropy and a multi-agent judge framework for efficiency.
Key contributions
- Introduction of a domain-agnostic framework for creativity evaluation of LLMs.
- Development of a novel metric for measuring divergent creativity using semantic entropy.
- Implementation of a multi-agent judge framework for context-sensitive evaluation of task fulfillment.
Notable insights
- The use of semantic entropy as a reference-free metric for measuring divergent creativity is a significant methodological advancement.
- The retrieval-based multi-agent judge framework enhances efficiency in evaluating convergent creativity, which could streamline assessments in various applications.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2606.11762v1 Announce Type: cross Abstract: Large language models (LLMs) have achieved remarkable progress in language understanding, reasoning, and generation, sparking growing interest in their creative potential. Realizing this potential requires systematic and scalable methods for evaluating creativity across diverse tasks. However, most existing creativity metrics are tightly coupled to specific tasks, embedding domain assumptions into the evaluation process, and limiting scalability and generality. To address this gap, we introduce an automated, domain-agnostic framework for quantifying LLM creativity across open-ended tasks. Our approach separates the measurement apparatus from the creative task itself, enabling scalable, task-agnostic assessment. Divergent creativity is measured using semantic entropy, a reference-free and robust metric for novelty and diversity, validated against human annotations, LLM-based novelty judgments and baseline diversity measures. Convergent creativity is assessed via a novel retrieval-based multi-agent judge framework that delivers context-sensitive evaluation of task fulfilment with over 60% improved efficiency. We validate our framework in three qualitatively distinct domains: problem-solving (MacGyver), research ideation (HypoGen), and creative writing (BookMIA), using a broad suite of LLMs. Empirical results show that our framework reliably captures key facets of creativity, including novelty, diversity, and task fulfilment, and reveal how model properties, such as size, temperature, recency, and reasoning, impact creative performance. Our work establishes a reproducible and generalizable standard for automated LLM creativity evaluation, paving the way for scalable benchmarking and accelerating progress in creative AI.