A Foundation Model for Multimodal Event Sequences in Financial Applications
Nikita Rusakov, Vladislav Meshkov, Konstantin Zorin, Gleb Zaripov, Alexander Uglov, Alexey Vasilev, Anton Klenitskiy
Why It Matters
What makes this one worth your time
This approach could streamline financial predictive modeling by leveraging diverse data sources, potentially leading to more accurate predictions and efficient model development.
A transformer model for multimodal financial data improves task performance and reduces development time.
Summary
The paper introduces a foundation transformer model for financial applications by pretraining on multimodal sequences of user events, integrating data from various sources into a unified sequence. This approach allows for early fusion of heterogeneous data and learning general-purpose representations, which are then used with engineered features for downstream tasks, resulting in improved performance and reduced development overhead.
Key contributions
- Development of a foundation transformer model for multimodal event sequences.
- Integration of diverse data sources into a unified chronological sequence.
- Demonstrated deployment in a major bank with measurable business improvements.
Notable insights
- Early fusion of heterogeneous data modalities can enhance the learning of general-purpose representations.
- Combining pretrained representations with engineered features can improve downstream task performance.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.09955v1 Announce Type: cross Abstract: Predictive modeling is a core component of modern financial services, where a wide range of tasks are traditionally addressed using separate models trained on manually engineered tabular features. This task-specific approach limits reuse and makes it difficult to fully exploit heterogeneous data sources such as transaction histories and digital interaction signals. In this paper, we present an approach based on pretraining a foundation transformer model on multimodal sequences of user events. Events from multiple data sources are unified into a single chronological sequence, enabling early fusion of heterogeneous modalities and learning of general-purpose representations via a next-event prediction objective. These representations are combined with existing engineered user features, on top of which lightweight neural models are trained for multiple downstream tasks. The proposed system outperforms traditional task-specific models while reducing development overhead. The approach was deployed in production at one of the biggest banks in Eastern Europe, resulting in measurable improvements in business metrics.