MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities
Mingqiao Ye, Zhaochong An, Zhitong Gao, Xian Liu, Fran\c{c}ois Fleuret, Chuan Li, Amir Zadeh, Serge Belongie, Afshin Dehghan, Jesse Allardice, David Mizrahi, O\u{g}uzhan Fatih Kar, Roman Bachmann, Amir Zamir
Why It Matters
What makes this one worth your time
This approach could simplify multimodal model architectures by eliminating the need for modality-specific components, potentially broadening the applicability of pre-trained models.
Modus enables symmetric any-to-any multimodal modeling using a decoder-only architecture.
Summary
The paper presents Modus, a decoder-only model for any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific components. It aims to leverage pre-trained decoder-only models and demonstrates competitive performance across various benchmarks.
Key contributions
- Introduction of a decoder-only architecture for any-to-any multimodal modeling.
- Demonstration of competitive performance with a single model across various benchmarks.
Notable insights
- The use of a decoder-only architecture allows for symmetric treatment of all modalities.
- Modus supports chained generation and cross-modal self-verification, enhancing its flexibility.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.25948v1 Announce Type: cross Abstract: Any-to-any models predict any modality from any combination of others within a single network, a formulation used in multimodal vision and vision-language models, and increasingly in scientific domains such as ecology and astronomy. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, impacting their performance and preventing them from using strong pre-trained decoder-only models as a prior. In this work, we investigate decoder-only any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. Because every modality is both an input and an output of the same model, the resulting model, named Modus, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model's own outputs with another generated modality. Modus demonstrates strong out-of-the-box performance and is competitive with specialist and multitask baselines using a single model across various benchmarks. All materials are open-sourced at https://modus-multimodal.epfl.ch/.