SkillVetBench: LLM-as-Judge for Multi-Dimensional Security Risk Evaluation in Open-Source LLM Agent Skills
Ismail Hossain, Sai Puppala, Md Jahangir Alam, Tanzim Ahad, Sajedul Talukder
Why It Matters
What makes this one worth your time
As open-source LLM agent ecosystems grow, ensuring the security of contributed skills is crucial to prevent malicious exploitation and data breaches.
SkillVetBench uses LLMs to semantically evaluate security risks in open-source LLM agent skills.
Summary
The paper introduces SkillVetBench, a system using LLMs to evaluate security risks in open-source LLM agent skills, addressing gaps in current code-based scanners by focusing on instruction-layer and multi-agent risks.
Key contributions
- Development of SkillVetBench, a public leaderboard for vetting agent skills using LLMs.
- Introduction of SARS, a five-dimensional agentic-risk metric for evaluating instruction-following systems.
- Demonstration of superior detection rates compared to conventional tools, particularly for instruction-layer threats.
Notable insights
- The use of LLMs as judges for semantic evaluation of security risks is a novel approach compared to traditional code-based scanners.
- The introduction of a multi-dimensional risk metric, SARS, provides a comprehensive evaluation framework for agentic risks.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.15899v1 Announce Type: cross Abstract: Open-source LLM agent ecosystems are growing rapidly, yet the security of community-contributed skills - modular tool definitions that extend agent capabilities - remains largely unvetted. The gap we fill: existing scanners operate at the code layer and are structurally blind to instruction-layer and multi-agent risk - natural-language directives that hijack an agent, exfiltrate data through encoded side channels, or chain harm across pipelines - so what is needed is a semantic, multi-dimensional vetting system rather than another signature matcher. We present SKILLVETBENCH, a live public leaderboard on Hugging Face that uses an LLM-as-Judge to vet agent skills. What is new: SARS (Skill Agentic Risk Score), a five-dimensional agentic-risk metric with a principled weighted formula for instruction-following systems. What is integrated: full CVSS v4.0 vector decomposition and a ClawHub dual-view that places our LLM-generated review beside the official marketplace verdict. What is demonstrated: drawing on our companion benchmark paper [ 1], the LLM-as-Judge stage achieves zero false negatives across 78 confirmed-malicious skills and zero false positives across 22 benign controls, while the best static baseline (SKILLSIEVE) still misses 15%; for instruction-layer categories such as Prompt Injection and Memory Poisoning, conventional tools miss between 89% and 100% of threats (e.g., CODEBERT detects none of nine memory-poisoning skills). Detection rates vary from 35% to 95% across four LLM evaluators, motivating ensemble scoring in production deployments.