Reasoning Consensus: Structural Ensembling of LLM Reasoning via Weighted DAG Aggregation
Amruta Parulekar, Jinu Lee, Dilek Hakkani-T\"ur, Hari Sundaram
Why It Matters
What makes this one worth your time
This work is relevant for AI researchers and engineers interested in enhancing the reliability and interpretability of LLM outputs, especially in high-stakes applications where understanding the reasoning process is crucial.
The paper introduces a method to ensemble LLM reasoning structures using weighted DAGs for improved accuracy and transparency.
Summary
The paper proposes a framework for ensembling reasoning structures of multiple large language models (LLMs) by using weighted Directed Acyclic Graphs (DAGs) to achieve 'Consensus Reasoning'. This method aims to improve reasoning accuracy and transparency by aggregating reasoning steps that are independently supported across different models. The framework is evaluated on six benchmarks, showing improved performance over a majority-vote baseline and providing an inspectable reasoning graph.
Key contributions
- Proposes a framework for ensembling reasoning structures of LLMs using weighted DAGs.
- Demonstrates improved performance over a majority-vote baseline across multiple benchmarks.
- Provides an inspectable consensus reasoning graph that enhances transparency.
Notable insights
- Using weighted DAGs to aggregate reasoning steps provides a novel way to enhance the interpretability of LLM outputs.
- The correlation between ensemble weights and LLM-judge rankings suggests a potential method for evaluating reasoning quality.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.27783v1 Announce Type: new Abstract: Large Language Models (LLMs) explore problems through chain-of-thought, but this exploration is buried in unstructured prose. On high-stakes tasks, users cannot tell which steps are well-supported, which alternatives were seriously considered, or how the final conclusion compares to those the model discarded. We propose a framework that ensembles the reasoning structure, not just the answers, of multiple LLMs by weighted merging of Directed Acyclic Graphs (DAGs) extracted from reasoning chains. We weight each step by how many traces independently attest to it, to return "Consensus Reasoning". Across six benchmarks spanning statutory interpretation, graduate-level science, narrative multi-hop reasoning, and first-order logic, our ensemble outperforms a matched-budget majority-vote baseline, with a maximum accuracy gain of 3.1% on MuSR-MM (narrative multi-hop reasoning). On a single model, the framework matches or exceeds self-consistency at the same trace budget while additionally exposing an inspectable consensus reasoning graph. Ensemble weights correlate with LLM-judge rankings of reasoning quality at Spearman $\rho = 0.30$-$0.51$, and consensus subgraphs are preferred over alternatives leading to the majority-vote answer in 54.4-65.4% of head-to-head comparisons across five of six datasets. We observe that our framework can also be used to analyze diverse reasoning perspectives for a problem.