Towards an Automated Test of LLM Security Knowledge
Shufan Chai, Liangliang Sun, Jessica Staddon
Why It Matters
What makes this one worth your time
As LLMs are increasingly deployed in security contexts, understanding their knowledge gaps is crucial for ensuring their reliability and effectiveness in real-world applications.
A novel method to evaluate LLM security knowledge using automated response analysis.
Summary
The paper presents a partially-automated method to assess large language models' (LLMs) knowledge of security topics by analyzing their responses to authoritative information from Consumer Protection Agencies, focusing on identity theft and impostor scams.
Key contributions
- Introduction of a partially-automated assessment method for LLM security knowledge.
- Application of the method to two specific security topics and multiple LLMs, demonstrating its utility.
- Identification of knowledge gaps in LLMs through analysis of their responses to security-related narratives.
Notable insights
- The use of authoritative sources like Consumer Protection Agencies to identify knowledge gaps in LLMs is a unique approach that may enhance the reliability of security assessments.
- The method's focus on response instability as an indicator of knowledge gaps provides a potentially scalable evaluation metric.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2607.18496v3 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used for a range of software, hardware and human-centered security tasks. Consequently, LLM performance on security tasks is an active area of measurement and research, often with a focus on identifying areas in which LLM security "knowledge" may be insufficient. Popular strategies for identifying LLM security knowledge gaps include building corpora of challenge questions or task benchmarks, strategies that require substantial manual work and security expertise to design and execute. We introduce a partially-automated method for assessing LLM knowledge of a security area. The method uses authoritative information from Consumer Protection Agencies (CPAs) to identify instability in LLM responses that can be indicative of knowledge gaps. We demonstrate the method for 2 security topics, identity theft and impostor scams, and 5 LLMs in 2 leading LLM families, Gemini and GPT, using publicly available information about identity theft and impostor scams from 6 CPAs. The method distinguishes between models that have and don't have sufficient knowledge to accurately identify the security topics in text narratives.