Metis: Memory Foundation Model
Zeyu Zhang, Ziliang Guo, Yihang Sun, Xichong Zhang, Xixuan Hao, Zehao Lin, Yang Zhang, Xiaoyan Zhao, Tong Shen, Bo Tang, Zhi-Qin John Xu, Junchi Yan, Haofen Wang, Xu Chen, Feiyu Xiong, Zhiyu Li, Tat-Seng Chua
Why It Matters
What makes this one worth your time
This work could lead to more efficient AI models with improved memory capabilities, reducing reliance on external memory modules and potentially enhancing performance in tasks requiring historical context.
Metis integrates native memory capabilities into foundation models for autonomous information storage and retrieval.
Summary
The paper introduces Metis, a memory foundation model that integrates native memory capabilities into foundation models, allowing them to store and utilize information autonomously. The model features a new architecture with a native memory state and uses memory attention to access historical information. The authors provide large-scale memory-specific training data and propose optimization objectives for mid-training. Metis's memory maintenance is gradient-free, requiring only a forward pass for updates.
Key contributions
- Introduction of a memory foundation model with native memory capabilities.
- Development of a new architecture with a native memory state.
- Proposal of memory-specific training data and optimization objectives.
Notable insights
- The use of memory attention to access compressed historical information within the model.
- Gradient-free online memory maintenance that simplifies the update process.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.26760v1 Announce Type: new Abstract: Recent advances in AI agents have increasingly internalized native capabilities into their underlying foundation models, giving rise to multimodal foundation models and large reasoning models. However, agent memory is still primarily implemented through external modules, leaving the native memory capability largely unexplored. In this paper, we take a first step toward this direction by introducing memory foundation models, which empower foundation models with native memory capabilities. We formalize native memory from two perspectives: a persistent and dynamically evolving memory state within the backbone, and native memory procedures that autonomously store and utilize information through model computation. We show that native memory offers advantages in architecture, end-to-end optimization, and efficiency. Based on this formulation, we propose Metis, the first prototype of memory foundation models. Metis introduces a new architecture that equips a foundation model with a native memory state, allowing historical information to be compressed into the model and accessed through memory attention. We construct large-scale memory-specific training data and introduce multiple optimization objectives to acquire these native memory procedures through mid-training. The online memory maintenance of Metis is gradient-free, and the memory update requires only a forward pass. At inference time, all learned model weights remain frozen, while the native memory states are autonomously transformed through standard forward computation. Through extensive experiments, we show that Metis exhibits native memory capabilities and further provide a detailed analysis of its strengths, limitations, and behaviors. To facilitate future research on memory foundation models, we release our project and model checkpoints.