Benchmark papers

Browse recent papers, open the full write-up, or filter the archive by topic.

236 papers found

ReviewPaperPublishedTopics

6.8Empirical Evaluation of Out-Of-Distribution Performance of Tabular Foundation ModelsUnderstanding the OOD performance of TFMs is crucial for their reliable deployment in real-world applications where distribution shifts are common.Jul 29, 2026Evaluation Benchmark Data

6.8Agentic AI for Scientific Reasoning in Autonomous Quantum Sensing ExperimentsThis research is relevant for AI engineers and researchers interested in the intersection of AI and quantum experiments, showcasing how autonomous agents can enhance scientific reasoning and experiment control.Jul 29, 2026LLM Agent Reasoning

7.2LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable EvaluationThis approach offers a scalable and potentially more objective method for evaluating LLMs, addressing issues with static benchmarks and subjective human preferences.Jul 29, 2026LLM Evaluation Benchmark

7.2Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural ContextThis framework is crucial for developing and assessing LLMs that are fair, safe, and accurate in the diverse Indian context, which is often overlooked by existing benchmarks.Jul 29, 2026LLM Evaluation Benchmark

7.2AI-generated Images Challenge Visual Trust in High-risk ScenariosUnderstanding the limitations of current detection systems in high-risk scenarios is crucial for developing more reliable AI tools to maintain visual trust and ensure public safety.Jul 28, 2026Vision Language Safety Benchmark

6.8AIR-BENCH Live: An Evolving Safety Benchmark for Foundation ModelsAs AI models and regulations rapidly evolve, a dynamic benchmark like AIR-BENCH Live helps ensure ongoing safety and compliance, which is crucial for developers and policymakers.Jul 28, 2026Benchmark Safety Evaluation

6.8Spatial Reasoning in LLM Game Agents: Impact of Causal Context and Multi-Step PlanningUnderstanding and improving spatial reasoning in LLM game agents can enhance their performance in complex tasks, which is crucial for developing more advanced AI systems.Jul 28, 2026LLM Agent Reasoning

6.8TLA$^{+}$-Bench: An Execution-Grounded Benchmark and Dataset for Natural-Language to TLA+ Specification GenerationThis work provides a more rigorous method for evaluating the correctness of formal specifications generated by language models, which is crucial for ensuring reliability in software engineering applications.Jul 28, 2026Benchmark Evaluation Coding

7.2SQBench: A Benchmark for Evaluating Task Delivery by Language-Model Agents in Production-Oriented WorkflowsThis benchmark addresses a gap in existing evaluations by focusing on verifiable deliverables, which is crucial for real-world applications of language models in production settings.Jul 28, 2026Benchmark Evaluation LLM

7.0Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and JudgingUnderstanding and improving multi-turn dialogue capabilities in LLMs is crucial for developing more effective conversational agents, and this benchmark provides a structured way to evaluate these capabilities.Jul 27, 2026LLM Evaluation Benchmark

7.2Atlas 2 -- Foundation models for clinical deploymentThis work is significant for AI researchers and engineers focused on healthcare, as it addresses key limitations in deploying pathology models clinically, potentially leading to more reliable and efficient diagnostic tools.Jul 27, 2026Vision Language Benchmark Data

6.8A Consensus-Based Framework for Relative Preference Evaluation of Large Language ModelsThis framework offers a scalable alternative for evaluating LLMs in scenarios with multiple valid answers, potentially aligning better with nuanced response quality assessments.Jul 27, 2026LLM Evaluation Benchmark

7.0ConfidenceBench: Evaluating Confidence Calibration in Large Language ModelsUnderstanding and improving confidence calibration in LLMs is crucial for deploying them in high-stakes environments where incorrect answers can have significant consequences.Jul 24, 2026LLM Evaluation Benchmark

7.0OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party SkillsUnderstanding and improving agent safety when using third-party skills is crucial for deploying reliable AI systems in real-world applications.Jul 24, 2026LLM Agent Safety

6.8Expectation Alignment of Language Models for Real-World User ExpectationsUnderstanding and aligning with real user expectations is crucial for improving the practical utility and user satisfaction of language models.Jul 24, 2026LLM Alignment Evaluation

6.5Benchmarking Large Language Models on Multi-Sensor Physical Hazard AssessmentUnderstanding the limitations of language models in multi-sensor environments is crucial for their deployment in safety-critical systems.Jul 24, 2026LLM Evaluation Safety

7.0It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of BeliefUnderstanding how linguistic framing affects LLM behavior can inform prompt engineering and improve model robustness.Jul 23, 2026LLM Evaluation Benchmark

6.8Self-Preference Bias in Rubric-Based Evaluation of Large Language ModelsUnderstanding and mitigating self-preference bias is crucial for fair and accurate evaluation of language models, which is essential for their development and deployment.Jul 23, 2026LLM Evaluation Benchmark

6.8Economic Evaluations of Language ModelsUnderstanding the economic impact of language models can guide their integration into the workforce, highlighting areas where they can save time and improve efficiency.Jul 23, 2026Evaluation Benchmark Infra

6.8Prompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM EvaluationUnderstanding prompt robustness is crucial for improving the reliability of language models in diverse applications, especially when interpreting their responses as indicative of beliefs or values.Jul 22, 2026LLM Evaluation Benchmark

6.5Saving the legacy of Hero Ibash: Evaluating Four Language Models for AminoacianUnderstanding how language models perform in low-resource languages can help bridge linguistic gaps and enhance NLP inclusivity.Jul 22, 2026Evaluation Benchmark

7.5When JSON Is Not Enough: Semantic Reliability of Schema-Constrained LLM Ordering AgentsUnderstanding the limitations of schema validation is crucial for developers relying on LLMs for safe and accurate transaction processing.Jul 22, 2026LLM Agent Evaluation

7.5FindStatBench: Evaluating Large Language Models on Combinatorial Code SynthesisThis benchmark provides a structured way to assess the capabilities of language models in generating code for complex combinatorial tasks, which is crucial for advancing AI's practical applications in programming.Jul 22, 2026LLM Evaluation Benchmark

7.5OpenMHC: Accelerating the Science of Wearable Foundation ModelsThis work addresses the critical lack of publicly available wearable health datasets, enabling researchers to advance the field of wearable health AI more effectively.Jul 22, 2026Data Benchmark

6.5Trusting sovereign language models as scientific instruments: evidence from Portugal's AMALIAUnderstanding the limitations of national language models like AMALIA in accurately annotating complex constructs is crucial for their effective deployment in real-world applications.Jul 21, 2026LLM Evaluation Data

7.2FETS Benchmark: Foundation Models Enable Scalable and Generalizable Energy Time Series ForecastingThis research highlights the potential of foundation models to provide scalable and generalizable solutions for energy forecasting, which is crucial for efficient energy system planning and operation.Jul 21, 2026Benchmark Data

7.2Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool ScenariosUnderstanding the limitations and capabilities of LLMs in generating complete software systems can guide future improvements in AI-driven software development.Jul 20, 2026LLM Benchmark Coding

6.8ToolAlignBench: Investigating Alignment Conflicts in Tool-Calling Enabled LLMsUnderstanding alignment conflicts in LLMs is crucial for deploying these models in sensitive environments, where conflicting values can lead to unpredictable and potentially harmful actions.Jul 18, 2026LLM Alignment Safety

7.5When Words Are Safe But Actions Kill: Probing Physical Danger Beyond Text Safety in Hidden-State Risk SpaceUnderstanding the difference between text-based safety and real-world risks is crucial for the safe deployment of AI systems in physical environments.Jul 18, 2026LLM Safety Evaluation

7.5Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code QualityUnderstanding language bias in code generation can lead to improved multilingual programming tools and practices, enhancing accessibility and usability for diverse developers.Jul 18, 2026Coding Evaluation Benchmark

7.2Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static BenchmarksThis approach can make LLM evaluation more scalable and cost-effective, addressing a critical bottleneck in AI research and development.Jul 16, 2026LLM Evaluation Benchmark

7.2SingGuard-NSFA: Extensible Guardrails for Agentic AI via Generative Reasoning and Real-Time ClassificationAs AI systems become more autonomous, ensuring their security against operational threats is crucial for safe deployment in real-world applications.Jul 16, 2026Agent Security Safety

6.5Exploring Zero-Shot Foundation Models for Multivariate Time Series Anomaly DetectionUnderstanding the limitations of zero-shot models in MTSAD can guide future research towards more effective anomaly detection techniques in critical domains.Jul 15, 2026Safety Benchmark

7.2First, do NOHARM: a medical safety benchmark and randomized study of physician and AI teaming on clinical consultationsUnderstanding and mitigating the risks of AI in clinical settings is crucial for ensuring patient safety and improving healthcare outcomes.Jul 15, 2026LLM Safety Benchmark

6.8What We Talk About When We Talk About LLM Planning: Evidence for Two Distinct Planning AbilitiesUnderstanding distinct planning abilities in LLMs can guide the development of more effective models and improve task-specific performance evaluation.Jul 14, 2026LLM Reasoning Evaluation

7.0Benchmarking the Robustness of Foundation Models for Mammography under Domain ShiftUnderstanding the robustness of foundation models in mammography under domain shifts is crucial for reliable clinical applications and improving diagnostic accuracy across diverse datasets.Jul 14, 2026Benchmark Evaluation Vision Language

7.0PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM AgentsUnderstanding the impact of multilinguality on LLM agents is crucial for developing more robust and versatile AI systems capable of handling real-world multilingual workflows.Jul 11, 2026LLM Agent Reasoning

6.8Institutional Red-Teaming: Deployment Rules, Not Just Models, Causally Shape Multi-Agent AI SafetyUnderstanding how deployment rules affect AI safety is crucial for designing systems that minimize risks in multi-agent environments.Jul 9, 2026Agent Safety Evaluation

6.8Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language ModelsUnderstanding how VLMs process spatial deictic expressions is crucial for improving their contextual reasoning and multilingual capabilities, which are essential for real-world applications.Jul 9, 2026Vision Language Evaluation Benchmark

7.0AI Chatbot Suicide Risk Detection and Response: Human Validation Study of the Open-Source VERA-MH Safety EvaluationAs AI chatbots become more prevalent in mental health support, ensuring their safety and reliability in high-risk situations like suicide risk detection is crucial for user trust and well-being.Jul 9, 2026LLM Evaluation Safety

6.8LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial ObservabilityUnderstanding the capabilities and limitations of LLMs in collaborative settings is crucial for developing more effective multi-agent systems and improving AI-human interactions.Jul 8, 2026LLM Agent Reasoning

7.5CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based ProgrammingThis research addresses the practical need for effective AI tutors in educational settings, particularly in K-12 environments, where privacy and cost are significant concerns.Jul 8, 2026LLM Evaluation Benchmark

7.5Prompt Framing Distorts Count-Based Evaluation of LLM Error Detection: Evidence from Numeric AnchoringUnderstanding the limitations of count-based metrics is crucial for developing more accurate evaluation methods for LLMs, impacting both research and practical applications in error detection.Jul 3, 2026Evaluation Benchmark

7.5A Unified Framework for the Evaluation of LLM Agentic CapabilitiesThis framework enables more reliable assessments of LLMs as agents, which is crucial for their deployment in safety-critical applications and for advancing research in AI capabilities.Jul 3, 2026LLM Agent Evaluation

7.8Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level EvaluationUnderstanding language sensitivity in AI evaluations can improve the design of benchmarks and enhance the reliability of AI systems in multilingual contexts.Jul 3, 2026Evaluation Benchmark LLM

7.5EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety GuardsThis research addresses the critical need for effective safety measures in embodied AI systems, which are increasingly deployed in real-world environments.Jul 2, 2026Evaluation Benchmark Vision Language

6.5Mapping the Evaluation Frontier: An Empirical Survey of the Bias-Reliability Tradeoff Across Eleven Evaluator-Agent ConditionsUnderstanding the bias-reliability tradeoff is crucial for improving the accuracy and reliability of LLM evaluation systems, which are fundamental in assessing AI models.Jul 2, 2026LLM Evaluation Benchmark

7.2Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral FrameworkUnderstanding LLM reasoning beyond correctness can lead to more reliable and context-aware AI applications, improving model selection and deployment strategies.Jul 2, 2026LLM Reasoning Evaluation

6.8Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground TruthUnderstanding and improving LLM performance in culturally nuanced and linguistically diverse contexts is crucial for deploying these models in real-world applications involving underrepresented languages.Jul 2, 2026LLM Evaluation Benchmark

6.8Comparing Large Language Models on Scrum Certification-Style Questions: Accuracy, Stability, and Error PatternsUnderstanding how LLMs perform on domain-specific certification questions can guide improvements in AI-driven educational tools and assessments.Jul 2, 2026LLM Evaluation Benchmark