# Distillation Matters: Empowering Sequential Recommenders to Match the Performance of Large Language Model

Source: [Distillation Matters: Empowering Sequential Recommenders to Match the Performance of Large Language Model](<https://www.sota2.com/research/paper/68fa74821e209131cea620f1>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T03:11:37.234Z

Authors: Yu Cui, Feng Liu, Pengbo Wang, Bohao Wang, Heng Tang, Yi Wan, Jun Wang, Jiawei Chen

Published: 2024-05-01T06:23:54.000Z

## Abstract

Owing to their powerful semantic reasoning capabilities, Large Language Models (LLMs) have been effectively utilized as recommenders, achieving impressive performance. However, the high inference latency of LLMs significantly restricts their practical deployment. To address this issue, this work investigates knowledge distillation from cumbersome LLM-based recommendation models to lightweight conventional sequential models. It encounters three challenges: 1) the teacher's knowledge may not always be reliable; 2) the capacity gap between the teacher and student makes it difficult for the student to assimilate the teacher's knowledge; 3) divergence in semantic space poses a challenge to distill the knowledge from embeddings. To tackle these challenges, this work proposes a novel distillation strategy, DLLM2Rec, specifically tailored for knowledge distillation from LLM-based recommendation models to conventional sequential models. DLLM2Rec comprises: 1) Importance-aware ranking distillation, which filters reliable and student-friendly knowledge by weighting instances according to teacher confidence and student-teacher consistency; 2) Collaborative embedding distillation integrates knowledge from teacher embeddings with collaborative signals mined from the data. Extensive experiments demonstrate the effectiveness of the proposed DLLM2Rec, boosting three typical sequential models with an average improvement of 47.97%, even enabling them to surpass LLM-based recommenders in some cases.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Sequential Recommendation on Amazon Office](<https://www.sota2.com/research/sota/sequential-recommendation-on-amazon-office.md>): Sequential Recommendation; Amazon Office; N@5: 1.66.

- [Sequential Recommendation on Amazon Book](<https://www.sota2.com/research/sota/sequential-recommendation-on-amazon-book.md>): Sequential Recommendation; Amazon-Book; N@5: 0.68.

- [Sequential Recommendation on Amazon Clothing](<https://www.sota2.com/research/sota/sequential-recommendation-on-amazon-clothing.md>): Sequential Recommendation; Amazon Clothing; N@5: 0.0028.

- [Sequential Recommendation on Amazon Books (test)](<https://www.sota2.com/research/sota/sequential-recommendation-on-amazon-books-test.md>): Sequential Recommendation; Books Amazon (test); NDCG@10: 0.0094.

- [Sequential Recommendation on Amazon Toy](<https://www.sota2.com/research/sota/sequential-recommendation-on-amazon-toy.md>): Sequential Recommendation; Amazon Toy; N@5: 0.8.

- [Sequential Recommendation on Amazon Office (test)](<https://www.sota2.com/research/sota/sequential-recommendation-on-amazon-office-test.md>): Sequential Recommendation; Amazon Office (test); NDCG@10: 2.31.

- [Sequential Recommendation on Amazon Toy (test)](<https://www.sota2.com/research/sota/sequential-recommendation-on-amazon-toy-test.md>): Sequential Recommendation; Amazon Toy (test); NDCG@10: 0.011.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2405.00338>)

[PDF](<https://arxiv.org/pdf/2405.00338.pdf>)
