Knowledge Graph Representations for LLM-Based Policy Compliance Reasoning
Wilder Baldwin, Sepideh Ghanavati
Why It Matters
What makes this one worth your time
As AI regulations become more critical, this research provides a method to improve compliance reasoning, which is essential for developing safe AI applications.
A novel approach to enhance LLMs' reasoning on AI policy compliance through knowledge graphs.
Summary
The paper presents a framework for constructing knowledge graphs from AI policy documents to facilitate policy compliance reasoning using large language models (LLMs), demonstrating improvements in performance across various reasoning tasks.
Key contributions
- Development of an agentic framework for knowledge graph construction from AI policy documents.
- Evaluation of five LLMs on a diverse set of policy QA tasks.
- Demonstration of KG augmentation benefits for LLM performance.
Notable insights
- The use of two ontology schemas for knowledge graph construction may reveal the adaptability of KGs to different regulatory contexts.
- The evaluation of LLMs across multiple reasoning types highlights the complexity of policy compliance tasks.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2604.27713v1 Announce Type: new Abstract: The risks posed by AI features are increasing as they are rapidly integrated into software applications. In response, regulations and standards for safe and secure AI have been proposed. In this paper, we present an agentic framework that constructs knowledge graphs (KGs) from AI policy documents and retrieves policy-relevant information to answer questions. We build KGs from three AI risk-related polices under two ontology schemas, and then evaluate five LLMs on 42 policy QA tasks spanning six reasoning types, from entity lookup to cross-policy inference, using both heuristic scoring and an LLM-as-judge. KG augmentation improves scores for all five models, and an open, LLM-discovered schema matches or exceeds the formal ontology.