Archon: A Unified Multimodal Model for Holistic Digital Human Generation
Chong Bao, Shichen Liu, Lijun Yu, David Futschik, Stylianos Moschoglou, Shefali Srivastava, Ziqian Bai, Feitong Tan, Guofeng Zhang, Zhaopeng Cui, Sean Fanello, Yinda Zhang
Why It Matters
What makes this one worth your time
This research addresses a significant challenge in digital human generation, which is crucial for applications in virtual reality, gaming, and interactive media.
Archon offers a comprehensive solution for generating digital humans across multiple modalities.
Summary
The paper introduces Archon, a unified multimodal model designed for holistic digital human generation, integrating seven modalities and addressing token explosion in high-fidelity video generation through innovative techniques.
Key contributions
- Development of a unified multimodal model that integrates seven modalities.
- Implementation of a memory-efficient semantic video reparameterization technique.
- Proposal of a novel method for decomposing cross-modal tasks to improve fidelity and controllability.
Notable insights
- The introduction of memory-efficient semantic video reparameterization significantly reduces token count while maintaining quality.
- The 'Thinking in Modality' approach allows for systematic handling of complex cross-modal tasks, enhancing overall model performance.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2605.30311v1 Announce Type: cross Abstract: Digital humans are fundamental to immersive interaction, yet creating a unified model for holistic modalities, including text, audio, motion, and visual content, remains an open challenge. In this paper, we present Archon, a fully pretrained, human-centric unified multimodal model for holistic avatar generation. Archon unifies seven modalities with modality-specific tokenizers, and a native autoregressive unified multimodal model pretrained on synchronized modalities and 72 diverse tasks to model holistic joint distributions. To address the token explosion challenge in high-fidelity talking videos, we introduce a memory-efficient semantic video reparameterization, achieving 4x token reduction while preserving fine-grained dynamics, coupled with a semantic-driven video diffusion decoder. We further propose a "Thinking in Modality" that decomposes ambiguous cross-modal tasks into stepwise thinking in an alternative chain of modality, progressively enhancing fidelity and controllability. Extensive experiments demonstrate that Archon achieves superior or comparable performance across diverse digital human generation tasks, validating the effectiveness of our unified framework. Project page: https://zju3dv.github.io/archon/.