Pre-trained Language Model and Knowledge Distillation for Lightweight Sequential Recommendation

• DOI: 10.48550/arxiv.2409.14810
• Publication Date: 2024-09-23
• Authors: Li Li; Mingyue Cheng; Zhiding Liu; Hao Zhang; Qi Liu; Enhong Chen
• Keywords: FOS: Computer and information sciences; Computer Science - Machine Learning; Information Retrieval (cs.IR); Computer Science - Information Retrieval; Machine Learning (cs.LG)

Sequential recommendation models user interests based on historical behaviors to provide personalized recommendation. Previous sequential algorithms primarily employ neural networks extract features of interests, achieving good performance. However, due the system datasets sparsity, these often small-scale network frameworks, resulting in weaker generalization capability. Recently, a series large pre-trained language have been proposed. Nonetheless, given real-time demands systems, challenge remains applying for rapid recommendations real scenarios. To address this, we propose algorithm model and knowledge distillation. The key proposed is transfer across domains achieve lightweight inference by operates two stages: first stage, fine-tune dataset task; second distill trained learned model. Extensive experiments multiple public show that enhances accuracy timely services.