LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendation
Shali Jiang, Hua Zheng, Boyang Liu, Laming Chen, Kenny Lov, Chuanqi Xu, Lisang Ding, Qinghai Zhou, Can Cui, Xiaolong Liu, Xiaoyi Liu, Yasmine Badr, Xin Xu, Jiyan Yang, Ellie Dingqiao Wen, Gerard Jonathan Mugisha Akkerhuis, Chenxiao Guan, Rong Jin, Ruichao Qiu, Xian Chen, Shifu Xu, Zhehui Zhou, Ping Chen, Rui Yang, Haicheng Chen, Xiangge Meng, Song Zhou, Dharak Kharod, Shuyu Xu, Qiang Jin, Qiao Yang, Wankun Zhu, Qin Huang, Yuzhen Huang, Darren Liu, Parish Aggarwal, Hui Zhou, Erzhuo Wang, Shuo Chang, Xiaorui Gan, Wenlin Chen, Santanu Kolay, Huayu Li
Why It Matters
What makes this one worth your time
This work addresses a critical limitation in knowledge distillation, potentially leading to more effective recommendation systems in large-scale applications.
LoopFM significantly boosts knowledge transfer from foundation models to compact models using historical representations.
Summary
The paper introduces LoopFM, a framework that enhances knowledge transfer from foundation models to compact vertical models by utilizing intermediate embeddings as input features, thereby improving performance without requiring real-time inference.
Key contributions
- Proposes a new framework (LoopFM) for knowledge transfer from foundation models to vertical models.
- Provides a theoretical framework with gain decomposition and transfer-ratio analysis.
- Demonstrates significant performance improvements on public benchmarks and industrial-scale systems.
Notable insights
- LoopFM's approach to structuring FM intermediate embeddings as input features is a novel method to enhance knowledge transfer.
- The framework's ability to operate without real-time FM inference could streamline deployment in industrial settings.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2605.29280v1 Announce Type: cross Abstract: Knowledge distillation (KD) transfers a single scalar prediction from a large foundation model (FM) to compact vertical models (VMs), suffering from diminishing transfer ratio -- the fraction of FM improvement captured by the VM -- as a single scalar cannot convey the rich intermediate knowledge that larger FMs learn. To address this bottleneck, we propose LoopFM (Learning frOm HistOrical ReP*resentations of FM), a framework that opens a high-bandwidth transfer channel by structuring FM intermediate embeddings as input features (e.g., user history sequence) for downstream VMs, without requiring real-time FM inference at serving and architectural coupling between FM and VM. We provide a theoretical framework for LoopFM with a gain decomposition and transfer-ratio analysis. On three public benchmarks, LoopFM demonstrates strong AUC improvements (e.g., 6\%+ on TaobaoAd) and complementary knowledge transfer capability with KD. On industrial-scale systems (billions of examples, trillion-parameter FMs), LoopFM approximately doubles the knowledge transfer ratio on top of KD, delivering a +0.5\% conversion improvement in Y1H1, and a +1.03\% and +1.22\% conversion improvement from two individual launches respectively in Y1H2.