Towards Foundation Models for Relational Databases with Language Models and Graph Neural Networks
Jingcheng Wu, Ratan Bahadur Thapa, Mojtaba Nayyeri, Lucas Etteldorf, Max Finkenbeiner, Fabian Leeske, Steffen Staab
Why It Matters
What makes this one worth your time
This work addresses a significant gap in deep learning for relational data, potentially improving predictive applications that rely on structured information.
A novel hybrid model enhances relational database representation using language models and graph neural networks.
Summary
The paper proposes a hybrid architecture that combines a fine-tuned BART encoder with a GraphSAGE-based GNN to enhance the representation of relational databases, demonstrating competitive performance on a specific task.
Key contributions
- Introduction of a hybrid architecture combining BART and GraphSAGE for relational data.
- Empirical evaluation on the RelBench dataset demonstrating competitive performance against existing models.
- Reduction of performance gap to state-of-the-art models, indicating the effectiveness of the proposed approach.
Notable insights
- The integration of GNNs with language models can capture both intra-row semantics and relational context, which is often overlooked in traditional approaches.
- The performance comparison with established baselines highlights the potential of hybrid models in bridging the gap between conventional methods and state-of-the-art solutions.
Possible limitations
- The abstract does not address the scalability of the proposed model to larger datasets or more complex relational structures.
- Performance metrics are limited to a specific task, which may not generalize across all relational database applications.
Abstract
arXiv:2605.16085v1 Announce Type: cross Abstract: Relational databases store much of the world's structured information, and they are essential for driving complex predictive applications. However, deep learning progress on relational data remains limited, as conventional approaches flatten databases into single tables via manual feature engineering, discarding relational context. Relational deep learning (RDL) addresses this by modeling databases as relational entity graphs (REGs) for graph neural networks (GNNs), but remains task- and database-specific. To combine the strengths of both paradigms, we propose a hybrid architecture combining a fine-tuned BART encoder to capture intra-row semantics with a GraphSAGE-based GNN over REGs to inject relational context. Experiments on RelBench show that the GNN substantially enriches BART's row embeddings, achieving a ROC-AUC of 67.40 on the driver-dnf task from the rel-f1 dataset. This performance is competitive with supervised baselines such as LightGBM (68.86) and narrows the gap to RDL (72.62) to within 5.22 points, though a substantial gap remains to state-of-the-art foundation models such as KumoRFM (82.63). These results suggest that lightweight hybrid LM-GNN architectures offer a promising and resource-efficient path towards foundation models for relational databases.