Benchmark papers

Browse recent papers, open the full write-up, or filter the archive by topic.

151 papers found

ReviewPaperPublishedTopics

7.0Multi-Modal Agents for Power Distribution Defect Detection: An Evaluation of Foundation ModelsUnderstanding how multimodal foundation models can be applied to industrial defect detection could enhance automation and reliability in critical infrastructure maintenance.Jun 12, 2026Agent Multimodal Evaluation

6.8Emerging Flexible Designs for Geospatial Multimodal Foundation ModelsUnderstanding the trade-offs in model architectures for geospatial data can help researchers and practitioners build more effective and flexible models for Earth observation tasks.Jun 12, 2026Multimodal Benchmark Architecture

7.5Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive ProgrammingUnderstanding LLMs' capabilities in educational contexts can enhance their integration into programming education and improve human learning outcomes.Jun 12, 2026LLM Evaluation Benchmark

7.0LLM-as-an-Investigator: Evidence-First Reasoning for Robust Interactive Problem DiagnosisThis research is relevant for AI engineers and researchers interested in improving the robustness and accuracy of LLMs in technical problem-solving contexts, particularly in reducing biases introduced by user assumptions.Jun 12, 2026LLM Agent Reasoning

7.5GrowLoop: Self-Evolving Conversation Evaluation Seeded by HumanAs large language models evolve, a robust and adaptive evaluation system is crucial for ensuring that conversational agents meet human-like standards, making this work relevant for both researchers and practitioners in AI.Jun 11, 2026Evaluation Benchmark LLM

6.8Are LLMs Bad at Moral Reasoning?Understanding the moral reasoning capabilities of LLMs is crucial for their safe deployment in dynamic environments where ethical considerations are important.Jun 11, 2026LLM Reasoning Evaluation

7.0The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMsUnderstanding how language models access cultural knowledge can improve their deployment in multilingual settings, ensuring more accurate and culturally relevant responses.Jun 10, 2026LLM Evaluation Benchmark

7.5Flaws in the LLM Automation NarrativeUnderstanding the limitations of LLMs in critical applications is essential for researchers and engineers to set realistic expectations and improve model evaluations.Jun 10, 2026LLM Evaluation Benchmark

7.5Does Capability Transfer to Subjective Behavior -- and Would Our Instruments Tell Us? A Self-Evolving, Trust-by-Construction Evaluation ParadigmThis approach addresses the challenges of subjective evaluation in LLMs, potentially leading to more reliable assessments of their behavioral capabilities.Jun 10, 2026Evaluation Benchmark

7.5BadRobot: Jailbreaking Embodied LLM Agents in the Physical WorldUnderstanding and mitigating the risks associated with embodied AI systems is crucial for ensuring their safe deployment in real-world applications.Jun 10, 2026LLM Agent Security

7.5RealMath-Eval: Why SOTA Judges Struggle with Real Human ReasoningUnderstanding the evaluation gap is crucial for improving LLMs and ensuring they can accurately assess diverse human reasoning in educational contexts.Jun 10, 2026LLM Evaluation Benchmark

6.8Mind the Gap: Can Frontier LLMs Pass a Standardized Office Proficiency Exam?Understanding the limitations of LLMs in practical office tasks is crucial for researchers and engineers aiming to improve automation capabilities and develop more effective AI systems.Jun 10, 2026LLM Evaluation Benchmark

6.8SLMJury: Can Small Language Models Judge as Well as Large Ones?Understanding the capabilities of smaller language models in evaluation tasks could lead to more cost-effective and scalable solutions in AI applications.Jun 9, 2026LLM Evaluation Benchmark

7.5LATTEArena: An Evaluation Framework for LLM-powered Tabular Feature Engineering (Extended Version)This framework enables researchers to systematically compare and improve LLM methods for tabular data, which is crucial for advancing automated data analysis.Jun 9, 2026Evaluation Benchmark

7.5Beyond English benchmarks: clinical llm evaluation in Brazilian PortugueseThis research addresses the critical gap in clinical AI evaluation for non-English languages, enhancing global accessibility and applicability of language models in healthcare.Jun 9, 2026LLM Evaluation Benchmark

6.8GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language ModelsUnderstanding the limitations of current LALMs in real-world, multilingual, and multicultural contexts is crucial for developing more effective and inclusive AI systems.Jun 9, 2026Multimodal Evaluation Benchmark

7.2Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMsUnderstanding the nuanced performance of code generation models across different languages and problem types can guide developers in selecting the right model for specific tasks and highlight areas for improvement in model development.Jun 9, 2026LLM Evaluation Benchmark

7.5TokaMind: A Multi-Modal Transformer Foundation Model for Tokamak Plasma DynamicsThis work provides a robust foundation for future research in fusion modeling, potentially accelerating advancements in plasma physics and fusion energy.Jun 8, 2026Multimodal Benchmark

7.2Phun-Bench: Evaluating LLMs on Phonological Understanding in ChineseUnderstanding LLMs' phonological capabilities is crucial for improving their linguistic performance, especially in languages like Chinese where phonology plays a key role.Jun 8, 2026Evaluation Benchmark

7.5Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ GenerationUnderstanding the limitations of LLMs in generating formal specifications is crucial for improving their reliability in industrial applications, particularly in verification tasks.Jun 6, 2026LLM Evaluation Benchmark

6.8Do More Agents Help? Controlled and Protocol-Aligned Evaluation of LLM Agent WorkflowsUnderstanding the efficiency and effectiveness of multi-agent systems in LLM workflows can guide the development of more cost-effective and accurate AI systems.Jun 6, 2026LLM Agent Evaluation

6.8Agent Memory: Characterization and System Implications of Stateful Long-Horizon WorkloadsUnderstanding and optimizing agent memory systems is crucial for improving the efficiency and scalability of long-horizon AI tasks.Jun 6, 2026LLM Agent Infra

7.0CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy BenchmarksThis framework allows researchers and engineers to evaluate language models for specific applications without the need for labeled data or concerns about benchmark contamination, potentially saving time and resources.Jun 6, 2026Evaluation Benchmark

7.5IatroBench: Pre-Registered Evidence of Iatrogenic Harm from AI Safety MeasuresUnderstanding how AI safety measures can inadvertently cause harm is crucial for developing more equitable and effective AI systems in healthcare.Jun 6, 2026Evaluation Safety Benchmark

7.5VGGSounder: Audio-Visual Evaluations for Foundation ModelsThis work is significant for researchers and engineers developing audio-visual models, as it provides a more reliable benchmark for assessing multi-modal understanding.Jun 5, 2026Multimodal Evaluation Benchmark

7.5Safety Under Scaffolding: How Evaluation Conditions Shape Measured SafetyUnderstanding the variability in safety measurements can help researchers and engineers make more informed decisions about deploying AI systems in real-world scenarios.Jun 5, 2026Safety Evaluation Benchmark

7.5NoRA: Evaluating Grounded Reasonableness in Visual First-person Normative Action ReasoningThis research addresses a critical gap in evaluating AI's normative competence in real-world scenarios, moving beyond simple action selection to deeper reasoning.Jun 5, 2026Agent Reasoning Multimodal

7.5FALSIFYBENCH: Evaluating Inductive Reasoning in LLMs with Rule Discovery GamesThis research provides insights into the reasoning capabilities of LLMs, which is crucial for their deployment in scientific tasks and understanding their limitations.Jun 5, 2026LLM Reasoning Evaluation

7.0AICompanionBench: Benchmarking LLMs-as-Judges for AI Companion SafetyUnderstanding and improving the safety of AI companion interactions is crucial as these platforms become more prevalent, and this work provides a tool for evaluating and enhancing model performance in this area.Jun 5, 2026LLM Evaluation Benchmark

7.2Cosmos 3: Omnimodal World Models for Physical AIThis work could streamline the development of embodied AI systems by providing a versatile, open-source framework that integrates multiple modalities, potentially accelerating research and application in Physical AI.Jun 3, 2026Multimodal Agent Vision Language

7.5Evaluating LLMs' Effectiveness on Real-World Consumer Device Repair QuestionsUnderstanding LLMs' performance in high-stakes domains like device repair is crucial for developing safer AI applications and improving user trust.Jun 3, 2026LLM Evaluation Benchmark

7.5Decomposing and Measuring Evaluation AwarenessUnderstanding evaluation awareness is crucial for improving the validity of benchmark results and ensuring that language models behave consistently across different evaluation contexts.Jun 3, 2026Evaluation Benchmark Alignment

7.2Business Utility of Large Language Models as Exploratory Data Analysis AgentsUnderstanding the limitations and capabilities of LLMs in EDA can guide businesses in adopting AI tools for data-driven decision-making.Jun 2, 2026LLM Agent Evaluation

6.8ROGUE: Misaligned Agent Behavior Arising from Ordinary Computer UseUnderstanding and addressing AI misalignment in everyday settings is crucial for ensuring the safe deployment of autonomous agents in real-world applications.Jun 2, 2026Agent Safety Alignment

7.5TravelEval: A Comprehensive Benchmarking Framework for Evaluating LLM-Powered Travel Planning AgentsThis framework could significantly enhance the evaluation of travel planning agents, leading to improved models and applications in real-world scenarios.Jun 2, 2026LLM Evaluation Benchmark

7.5Social Reasoning in Machines: Investigating Collective Truth-Seeking Dynamics in Large Language Model DebateUnderstanding how collective reasoning can improve AI performance has implications for developing more reliable and effective AI systems in various applications.Jun 1, 2026LLM Reasoning Evaluation

6.8EUDAIMONIA: Evaluating Undesirable Dynamics in AIUnderstanding and mitigating the social risks of LLMs is crucial for ensuring user welfare and ethical AI deployment in real-world applications.Jun 1, 2026LLM Alignment Evaluation

7.0LH-Bench: Skill-Grounded Evaluation of Long-Horizon Agents on Subjective Enterprise TasksThis work is relevant for AI researchers and engineers interested in developing and evaluating AI systems that operate in complex, subjective, and context-dependent enterprise environments.Jun 1, 2026LLM Agent Evaluation

6.8Relevance as a Vulnerability: How Web Retrieval Degrades Safety Alignment in LLM AgentsUnderstanding and mitigating the safety risks associated with web retrieval in LLM agents is crucial for developing secure AI systems that can leverage external information effectively.May 29, 2026LLM Agent Safety

8.5Thinking Fast, Thinking Wrong: Intuitiveness Modulates LLM Counterfactual Reasoning in Policy EvaluationUnderstanding how LLMs handle counterfactual reasoning is crucial for their application in real-world policy contexts, where intuitive biases can lead to misinterpretations of complex data.May 29, 2026Benchmark Evaluation LLM

8.5Revisiting the Reliability of Language Models in Instruction-FollowingAs language models are increasingly deployed in real-world applications, ensuring their reliability across varied and nuanced user inputs is crucial for dependable and trustworthy AI services.May 29, 2026Benchmark Evaluation LLM

7.0MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMsUnderstanding and improving the social and strategic reasoning of LLMs is crucial for their deployment in real-world multi-agent scenarios, making this evaluation platform a valuable tool for researchers and developers.May 29, 2026LLM Agent Reasoning

6.8GPF-LiveNews: A Streaming Evaluation Protocol for Group-Conditioned Framing in Large Language ModelsUnderstanding how language models frame information for different audiences is crucial for ensuring fairness and mitigating bias in AI systems.May 29, 2026LLM Evaluation Benchmark

6.8A Multi-dimensional Framework for Evaluating Generalization in EEG Foundation ModelsUnderstanding EEG model performance in low-resource settings is crucial for practical applications in neurotechnology and clinical fields.May 28, 2026Evaluation Benchmark Other

7.2FLORO: A Multimodal Geospatial Foundation Model for Ecological Remote Sensing Across Sensors and ScalesThis work is relevant for AI researchers and engineers working on remote sensing applications, as it demonstrates effective model performance with smaller datasets, which is crucial for ecological and environmental monitoring.May 28, 2026Multimodal Data Benchmark

6.8When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated EvaluationUnderstanding and mitigating self-bias in LLM evaluations is crucial for developing fair and reliable AI systems, especially as automated benchmarking becomes more prevalent.May 27, 2026LLM Evaluation Benchmark

7.0Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language ModelsThis approach could lead to more nuanced and informative evaluations of language models, highlighting specific capability gaps and improving model development.May 27, 2026LLM Evaluation Benchmark

8.2DIANOIA: Diagnostic Decomposition and Joint Optimization for Multi-Agent ReasoningThis framework enables practitioners to make informed design choices for multi-agent systems, improving efficiency and effectiveness in various tasks.May 27, 2026Agent Reasoning Evaluation

6.8Position: AI Safety Requires Effective ControllabilityEnsuring AI systems can be controlled and overridden in real-time is crucial for deploying them safely in dynamic environments.May 27, 2026Safety Alignment Benchmark

6.8Beyond Questions: Evaluating What Large Language Models (Actually) KnowUnderstanding the full scope of what LLMs know can improve their deployment in real-world applications by ensuring more comprehensive and unbiased knowledge assessments.May 27, 2026LLM Evaluation Benchmark