LLM-Guided Planning for Multi-hop Reasoning over Multimodal Nuclear Regulatory Documents
Mingyu Jeon, Bokyeong Kim, Suwan Cho, Jae Young Suh, Yonggyun Yu
Why It Matters
What makes this one worth your time
This research is relevant for AI engineers and researchers interested in improving document analysis and reasoning capabilities, particularly in complex and high-stakes domains like nuclear regulation.
LLM-guided planning enhances multi-hop reasoning over complex nuclear documents.
Summary
The paper presents a system for multi-hop reasoning over nuclear regulatory documents using a large language model (LLM) as a planning agent. The system dynamically constructs a knowledge graph and operates over a vectorless document tree to improve accuracy in evidence gathering. It achieves 81.5% accuracy on a benchmark and outperforms several existing methods.
Key contributions
- Development of an LLM-based planning agent for document analysis.
- Demonstrated superior accuracy over existing methods in a specific benchmark.
- Integration of a traceability module for auditing inferred edges.
Notable insights
- The use of a vectorless document tree and dynamic knowledge graph for state management in multi-hop reasoning tasks.
- Significant performance improvement through state-conditioned action selection in planning.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.29399v1 Announce Type: new Abstract: Reviewing nuclear regulatory documents requires multi-hop reasoning across tens of thousands of pages, where judgments depend on evidence assembled across multiple chapters. We frame this task as planning: an LLM-based agent observes the evidence collected so far, picks the next document fragment to inspect, and stops when the evidence is sufficient. The agent operates over a vectorless document tree using browse, read, and search tools, and maintains a dynamic knowledge graph (KG) as state. On a 200-question benchmark over NuScale Final Safety Analysis Report (FSAR) documents, the system reaches 81.5% accuracy with a RAGAS Faithfulness of 0.93. The dominant performance factor is planning: against PageIndex, which uses the same document tree without state-conditioned action selection, the gap is +38.0pp (43.5% to 81.5%, p<0.001). The system also outperforms LightRAG (73.0%, p<0.05), HippoRAG (70.5%, p<0.01), and GraphRAG (49.5%, p<0.001), and matches RAPTOR (75.5%, p=0.11) without offline indexing. Edge inference adds 2.8x cost without raising accuracy; we retain it as a traceability module. Of 7,391 inferred edges, 3 Violates edges (0.04%) flag scope boundaries (Q058) and partial conformance (Q176) as typed annotations that a human reviewer can audit.