CHARM: A Multimodal Graph Foundation Model with Hierarchical Context Modeling for Zero-Shot Transfer
Ankang Yang, Jitao Zhao, Di Jin, Yuxiao Huang, Dongxiao He
Why It Matters
What makes this one worth your time
This research addresses the critical challenge of adapting models to new graph domains without extensive labeling, which is a common limitation in real-world applications.
CHARM enables effective zero-shot transfer in multimodal graph tasks through hierarchical context modeling.
Summary
The paper introduces CHARM, a multimodal graph foundation model designed for zero-shot transfer by employing hierarchical context modeling to enhance knowledge transfer across different graph domains without requiring target-domain fine-tuning.
Key contributions
- Development of CHARM, a multimodal graph foundation model.
- Introduction of hierarchical context modeling to improve zero-shot transfer.
- Creation of a modality-aware graph context encoder that integrates multimodal information with graph structure.
Notable insights
- The use of hierarchical graph contexts to capture multimodal semantics and cross-modal relations is a novel approach that could enhance generalization across domains.
- Integrating a modality-aware graph context encoder with a large language model represents an innovative fusion of graph and language processing techniques.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2607.26023v1 Announce Type: new Abstract: Graph foundation models (GFMs) have emerged as a promising paradigm for transferring knowledge across graph domains and tasks. Real-world graphs associate nodes with text, images, and other modalities, making multimodal graphs essential for representing complex entities and relations. Moreover, collecting labels and adapting models for every new graph domain is costly and often infeasible, motivating zero-shot transfer. Unfortunately, zero-shot transfer on multimodal graphs remains underexplored. Existing GNN-based graph foundation models typically require downstream adaptation, whereas LLM-based graph methods mainly address unimodal graphs or tasks within a single domain. This setting presents two key challenges. First, models must generalize knowledge from individual modalities while capturing transferable cross-modal relations. Second, without target-domain fine-tuning, node representations remain entangled with domain-specific structures and modality-specific characteristics, obscuring shared concepts in unseen domains. To address these challenges, we propose CHARM, a multimodal graph foundation model with hierarchical context modeling for zero-shot transfer. CHARM replaces isolated raw nodes with hierarchical graph contexts that capture multimodal semantics and cross-modal relations. These contexts map domain-specific node patterns to shared high-level concepts, reducing reliance on target-domain supervision or adaptation. A modality-aware graph context encoder integrates multimodal information with graph structure and converts the resulting representations into graph tokens for a large language model . Experiments show consistent improvements on zero-shot multimodal graph tasks.