Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing
Feng Wang, Canmiao Fu, Zhipeng Huang, Chen Li, Jing Lyu, Ge Li
Why It Matters
What makes this one worth your time
This research addresses critical limitations in existing multimodal models, offering a scalable solution for complex interactions that could significantly enhance applications in AI-driven dialogue systems.
A novel multimodal agent that improves long-horizon dialogue through structured memory and efficient retrieval.
Summary
The paper introduces a Cognitive-structured Multimodal Agent that enhances long-horizon multimodal dialogue by externalizing visual information into an Episodic Visual Memory, allowing for selective reactivation of relevant episodes, and presents a new benchmark for evaluating episodic visual recall.
Key contributions
- Introduction of a Cognitive-structured Multimodal Agent with a Perceptual Abstraction Engine and Cognitive Retrieval Engine.
- Development of a Unified Scenario Engine for generating structured multi-turn conversations with retrieval annotations.
- Creation of a long-horizon visual-dialogue benchmark for evaluating episodic visual recall.
Notable insights
- The use of an Episodic Visual Memory allows for more efficient handling of visual information over extended dialogues, reducing the burden of visual token explosion.
- The development of a Unified Scenario Engine for generating structured conversations with retrieval annotations is a clever approach to tackle the lack of turn-level retrieval supervision.
Possible limitations
- Potential challenges in the scalability of the Episodic Visual Memory for very large datasets or real-time applications.
- Not stated in the abstract.
Abstract
arXiv:2607.08497v1 Announce Type: cross Abstract: Recent unified multimodal models show a single architecture can jointly perform vision/language understanding and image generation/editing. However, they repeatedly feed all historical visual and textual inputs into a shared context window, limiting long-horizon multimodal dialogue due to visual token explosion and unreliable cross-turn referencing. We propose a Cognitive-structured Multimodal Agent that externalizes visual information into an Episodic Visual Memory and selectively reactivates relevant episodes during reasoning. The agent consists of a Perceptual Abstraction Engine for structured visual abstraction, a Cognitive Retrieval Engine for cross-turn memory retrieval, and a Multimodal Executive Controller for autonomous task inference and action planning. To address the lack of turn-level retrieval supervision in existing datasets, we develop a Unified Scenario Engine that programmatically generates structured multi-turn conversations with fine-grained retrieval annotations, enabling reinforcement learning to optimize abstraction and retrieval policies. We also construct a long-horizon visual-dialogue benchmark stratified by difficulty to evaluate episodic visual recall. Our 8B agent achieves 91.4% retrieval accuracy over 20-turn sessions, surpassing 32B baselines by +8.2% while nearly halving per-turn inference time (23.1s -> 12.7s). We further present the Cognitive-structured Multimodal Agent Harness (CMA-Harness), a tool-augmented deployment of the same cognitive structure integrating persistent multimodal memory, web access, image generation/editing/composition tools, and OpenAI-compatible serving. Structured memory and modular decision-making offer a more scalable, efficient paradigm for long-horizon multimodal agents than monolithic parameter scaling. Code: https://github.com/caseclose/cma-harness ; Project page: https://caseclose.github.io/cma-harness/