Latent Memory Palace: Reasoning for Control as Autoregressive Variational Inference
Chuning Zhu, Eva Xu, Jose Barreiros, Krishnan Srinivasan, Paarth Shah, Abhishek Gupta
Why It Matters
What makes this one worth your time
This work addresses the challenge of transferring adaptive reasoning capabilities from language models to continuous control, which is crucial for advancing AI applications in robotics and autonomous systems.
Latent Memory Palace offers a novel approach to reasoning in continuous control through autoregressive latent inference.
Summary
The paper introduces the Latent Memory Palace (LMP), a method that organizes information in an autoregressive latent space to enhance reasoning for control policies, achieving strong performance in both simulations and real-world applications.
Key contributions
- Introduction of the Latent Memory Palace (LMP) framework for reasoning in control policies.
- Development of a latent-space reinforcement learning technique for optimizing variational inference.
- Creation of a variable-length action tokenizer (LMP-tok) that improves downstream autoregressive policy performance.
Notable insights
- The use of a memory palace analogy for organizing latent information is a creative approach that may enhance retrieval efficiency.
- The derivation of a latent-space reinforcement learning technique to optimize variational lower bounds is a significant methodological contribution.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2607.08724v1 Announce Type: new Abstract: Human decision-making is highly flexible -- some actions are taken immediately; others require longer deliberation. Language models have exhibited a similar capacity for adaptive "reasoning." However, transferring this capability to continuous control policies has been challenging, as directly reasoning in language space may lack the granularity for spatial understanding and precise motions. In this work, we show that reasoning for control policies can emerge by organizing information in an autoregressive latent space reminiscent of a memory palace, where retrieval is iterative and adaptive. Our method, Latent Memory Palace (LMP), formulates reasoning as variational inference with an autoregressive latent distribution. We derive a latent-space reinforcement learning technique to tractably optimize its variational lower bound. The resulting policy, LMP-$\pi$, achieves strong empirical performance in simulation and real-world domains while exhibiting interpretable, adaptive allocation of test-time compute. We further show that the same framework yields a variable-length action tokenizer, LMP-$\texttt{tok}$, which significantly improves the performance of downstream autoregressive policies. Together, these results present a new perspective on latent reasoning for control through the lens of variational inference.