Toward Federated Multimodal Graph Foundation Models: A Topology-Aware Multimodal Alignment Framework
Xunkai Li, Guohao Fu, Yuming Ai, Zhengyu Wu, Hongchao Qin, Rong-Hua Li, Guoren Wang
Why It Matters
What makes this one worth your time
This work is significant for AI engineers and researchers interested in privacy-preserving collaborative learning across multimodal datasets, offering a method to improve model performance without compromising data privacy.
FedGAMMA advances federated learning for multimodal graphs by aligning semantic and structural data across silos.
Summary
The paper introduces FedGAMMA, a framework for federated multimodal graph learning that aligns semantic and structural information across privacy-restricted silos. It employs a two-stage process involving federated pre-training and prompt-based fine-tuning, using techniques like optimal transport and dual-channel affinity-aware aggregation to enhance model performance without exposing raw data.
Key contributions
- Introduction of FedGAMMA, a federated learning framework for multimodal graphs.
- A topology-aware graph fusion module that decouples semantic and structural views.
- A prompt-based fine-tuning mechanism that adapts pretrained encoders using lightweight graph-aware prompts.
Notable insights
- The use of optimal transport for cross-modal alignment in a federated setting is a clever approach to handle semantic differences.
- The dual-channel affinity-aware aggregation mechanism allows for estimating client similarity without exposing raw data.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.15687v1 Announce Type: new Abstract: Multimodal-attributed graphs (MAGs), whose nodes carry modalities such as images and text alongside topological structure, now pervade applications including social platforms, e-commerce, and biomedical networks, offering richer semantic signals than single-modality graphs. In practice, such graphs are fragmented across privacy-restricted silos owned by different platforms and institutions, so learning a broadly transferable model over them demands collaborative training that never exposes raw data. This places the task at the intersection of multimodal graph learning and federated learning, yet existing methods cover only one side of it. To address the challenges from these two perspectives, we propose FedGAMMA, casting federated multimodal graph foundation learning as a two-stage semantic-structural alignment problem of federated pre-training and prompt-based fine-tuning. During pre-training, a shared-private semantic enhancer disentangles cross-modal commonality from modality-specific information, aligning it through optimal transport, a topology-aware graph fusion module decouples semantic and structural views via semantic residual graphs and dual positional encodings, and a dual-channel affinity-aware aggregation mechanism estimates client similarity from feature and graph centroids without exposing raw data. During fine-tuning, FedGAMMA adapts the pretrained encoder through lightweight graph-aware prompts, a shared prompt pool with controlled exploration, and channel-wise prompt synchronization. Experiments on twelve multimodal graph datasets show FedGAMMA consistently surpassing a broad range of baselines across downstream tasks, with gains of up to 12.96%. FedGAMMA further outperforms competitive baselines accross multi-domain datasets on multiple tasks with up to 5.71% under few-shot learning scenario.