Conversational Recommendation on REDDIT V2 (test)
11.46Recall@5CRAG
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| CRAGevaluation_setting=off-policy2025.10 | 11.46 | 17.15 | 20.3 | 22.12 | 8.85 | 10.65 | 11.64 | 12.27 | |
| GPT-4omode=zero-shot, evaluation_setting=off-policy2025.10 | 11.06 | 16.25 | 19.92 | 21.47 | 8.61 | 10.28 | 11.36 | 11.97 | |
| Llama-3.2-3B + SFT + Rank-GRPO (exp_inf)backbone=Llama-3.2-3B, training_protocol=SFT + Rank-GRPO, loss_variant=exp_inf, evaluation_setting=off-policy2025.10 | 10.92 | 16.53 | 20.1 | 22.72 | 8.45 | 10.28 | 11.32 | 12.03 | |
| Llama-3.2-1B + SFT + Rank-GRPO (exp_inf)backbone=Llama-3.2-1B, training_protocol=SFT + Rank-GRPO, loss_variant=exp_inf, evaluation_setting=off-policy2025.10 | 10.37 | 15.55 | 19.27 | 21.66 | 8.06 | 9.73 | 10.81 | 11.46 | |
| Llama-3.2-3B + SFT + Rank-GRPO (log)backbone=Llama-3.2-3B, training_protocol=SFT + Rank-GRPO, loss_variant=log, evaluation_setting=off-policy2025.10 | 10.37 | 15.72 | 19.58 | 22.2 | 8.16 | 9.89 | 11.01 | 11.69 | |
| Llama-3.2-3B + SFT + Vanilla GRPObackbone=Llama-3.2-3B, training_protocol=SFT + Vanilla GRPO, evaluation_setting=off-policy2025.10 | 9.52 | 14.25 | 17.26 | 19.52 | 7.66 | 9.16 | 10.04 | 10.65 | |
| GPT-4o-minimode=zero-shot, evaluation_setting=off-policy2025.10 | 9.49 | 13.48 | 16 | 16.87 | 7.47 | 8.77 | 9.5 | 9.73 | |
| Llama-3.2-3B + SFT + GSPObackbone=Llama-3.2-3B, training_protocol=SFT + GSPO, evaluation_setting=off-policy2025.10 | 9.42 | 14.36 | 17.7 | 19.43 | 7.58 | 9.14 | 10.11 | 10.58 | |
| Llama-3.2-1B + SFT + Vanilla GRPObackbone=Llama-3.2-1B, training_protocol=SFT + Vanilla GRPO, evaluation_setting=off-policy2025.10 | 9.37 | 14.15 | 17.26 | 19.74 | 7.28 | 8.82 | 9.75 | 10.42 | |
| Llama-3.2-1B + SFT + Rank-GRPO (log)backbone=Llama-3.2-1B, training_protocol=SFT + Rank-GRPO, loss_variant=log, evaluation_setting=off-policy2025.10 | 9.37 | 14.64 | 18.32 | 20.66 | 7.31 | 9.02 | 10.11 | 10.74 | |
| Qwen2.5-0.5B + SFT + Rank-GRPO (exp_inf)backbone=Qwen2.5-0.5B, training_protocol=SFT + Rank-GRPO, loss_variant=exp_inf, evaluation_setting=off-policy2025.10 | 9.35 | 14.12 | 17.38 | 18.99 | 7.35 | 8.87 | 9.82 | 10.25 | |
| Llama-3.2-1B + SFT + GSPObackbone=Llama-3.2-1B, training_protocol=SFT + GSPO, evaluation_setting=off-policy2025.10 | 9.01 | 13.89 | 16.93 | 18.99 | 7 | 8.57 | 9.47 | 10.02 | |
| Qwen2.5-0.5B + SFT + Rank-GRPO (log)backbone=Qwen2.5-0.5B, training_protocol=SFT + Rank-GRPO, loss_variant=log, evaluation_setting=off-policy2025.10 | 8.18 | 12.56 | 16.04 | 17.64 | 6.44 | 7.84 | 8.86 | 9.3 | |
| Qwen2.5-0.5B + SFT + GSPObackbone=Qwen2.5-0.5B, training_protocol=SFT + GSPO, evaluation_setting=off-policy2025.10 | 8.16 | 12.43 | 15.11 | 16.41 | 6.47 | 7.84 | 8.62 | 8.96 | |
| NBCRSevaluation_setting=off-policy2025.10 | 8.01 | 12.9 | 16.55 | 20.19 | 6.61 | 8.33 | 9.54 | 10.48 | |
| Llama-3.2-3B + SFTbackbone=Llama-3.2-3B, training_protocol=SFT, evaluation_setting=off-policy2025.10 | 7.88 | 11.91 | 14.1 | 15.41 | 6.15 | 7.44 | 8.07 | 8.42 | |
| Qwen2.5-0.5B + SFT + Vanilla GRPObackbone=Qwen2.5-0.5B, training_protocol=SFT + Vanilla GRPO, evaluation_setting=off-policy2025.10 | 7.81 | 11.76 | 14.29 | 15.38 | 6.23 | 7.48 | 8.23 | 8.52 | |
| Llama-3.2-1B + SFTbackbone=Llama-3.2-1B, training_protocol=SFT, evaluation_setting=off-policy2025.10 | 7.54 | 11.48 | 13.54 | 14.98 | 5.95 | 7.23 | 7.82 | 8.19 | |
| UniCRSevaluation_setting=off-policy2025.10 | 7.22 | 10.53 | 13.44 | 14.94 | 5.48 | 6.4 | 7.26 | 7.78 | |
| Qwen2.5-0.5B + SFTbackbone=Qwen2.5-0.5B, training_protocol=SFT, evaluation_setting=off-policy2025.10 | 6.42 | 10.27 | 12.12 | 13.08 | 5.02 | 6.25 | 6.78 | 7.04 | |
| KGSFevaluation_setting=off-policy2025.10 | 5.79 | 9.21 | 12.46 | 14.33 | 4.05 | 5.03 | 5.99 | 6.62 | |
| Llama-3.2-3Bbackbone=Llama-3.2-3B, mode=zero-shot, evaluation_setting=off-policy2025.10 | 5.74 | 8.05 | 9.12 | 9.61 | 4.63 | 5.35 | 5.66 | 5.8 | |
| KBRDevaluation_setting=off-policy2025.10 | 4.44 | 8.13 | 10.58 | 12.23 | 3.05 | 4.18 | 4.9 | 5.45 | |
| Llama-3.2-1Bbackbone=Llama-3.2-1B, mode=zero-shot, evaluation_setting=off-policy2025.10 | 1.87 | 2.4 | 2.57 | 2.63 | 1.57 | 1.7 | 1.75 | 1.76 | |
| Redialevaluation_setting=off-policy2025.10 | 1.03 | 2.28 | 2.74 | 3.22 | 0.73 | 1.13 | 1.28 | 1.43 | |
| Qwen2.5-0.5Bbackbone=Qwen2.5-0.5B, mode=zero-shot, evaluation_setting=off-policy2025.10 | 0.21 | 0.21 | 0.21 | 0.21 | 0.23 | 0.22 | 0.21 | 0.21 |