E-commerce Shopping Assistance on SmartShopBench 1.0 (test 120 queries)
93.35Precision (PC)ChatShopBuddy-SFT-RL
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| ChatShopBuddy-SFT-RLThinking Model Category=Thinking Model, # Tokens=6332026.03 | 93.35 | 99.6 | 84.97 | 75.22 | 34.2 | 0.6325 | 0.0096 | |
| GPT-5.2Thinking Model Category=Thinking Model, # Tokens=N/A2026.03 | 88.15 | 99.8 | 99.2 | 75.85 | 42.5 | 0.675 | 0.0058 | |
| ChatShopBuddy-SFTThinking Model Category=Thinking Model, # Tokens=1,2002026.03 | 86.88 | 98.55 | 72.28 | 60.4 | 18.3 | 0.48 | 0.0606 | |
| DeepSeek-V3.2-reasonerThinking Model Category=Thinking Model, # Tokens=1,4122026.03 | 86.05 | 99.8 | 79.38 | 62.1 | 19.2 | 0.62 | 0.0271 | |
| Gemini-3-Pro-PreviewThinking Model Category=Thinking Model, # Tokens=N/A2026.03 | 80.4 | 99.8 | 70.85 | 55.03 | 15.8 | 0.6725 | 0.005 | |
| GPT-5.2(think off)Thinking Model Category=Non-Thinking Model, # Tokens=-2026.03 | 80.03 | 99.8 | 91.88 | 66.88 | 35 | 0.6925 | 0.015 | |
| Doubao-seed-1.8Thinking Model Category=Thinking Model, # Tokens=1,0562026.03 | 76.47 | 99.38 | 58.75 | 39.77 | 3.3 | 0.2825 | 0.0189 | |
| Gemini-3-Flash-Preview(minimal)Thinking Model Category=Non-Thinking Model, # Tokens=-2026.03 | 74.8 | 100 | 71.22 | 48.75 | 9.2 | 0.56 | 0.0163 | |
| Doubao-Seed-1.8(think off)Thinking Model Category=Non-Thinking Model, # Tokens=-2026.03 | 73.53 | 98.75 | 49.38 | 30.2 | 1.7 | 0.3525 | 0.0222 | |
| Kimi-K2.5Thinking Model Category=Thinking Model, # Tokens=5152026.03 | 70.42 | 98.95 | 77.72 | 47.73 | 12.5 | 0.49 | 0.0115 | |
| DeepSeek-V3.2-chatThinking Model Category=Non-Thinking Model, # Tokens=-2026.03 | 69.8 | 98.75 | 66.9 | 43.33 | 7.5 | 0.455 | 0.0058 | |
| GLM-4.7Thinking Model Category=Thinking Model, # Tokens=7642026.03 | 64.8 | 99.6 | 65.4 | 37.5 | 2.5 | 0.425 | 0.01 | |
| GLM-4.7(think off)Thinking Model Category=Non-Thinking Model, # Tokens=-2026.03 | 61.67 | 99.6 | 67.9 | 38.98 | 5 | 0.4825 | 0.0096 | |
| Qwen3-30B-A3B-Thinking-2507Thinking Model Category=Thinking Model, # Tokens=2,8032026.03 | 47.3 | 98.55 | 31.7 | 12.7 | 0 | 0.6275 | 0.0096 | |
| Qwen3-235B-A22B-Instruct-2507Thinking Model Category=Non-Thinking Model, # Tokens=-2026.03 | 42.3 | 99.4 | 65.42 | 25.2 | 2.5 | 0.5775 | 0.0126 | |
| Qwen3-235B-A22B-Thinking-2507Thinking Model Category=Thinking Model, # Tokens=1,5262026.03 | 36.88 | 99.4 | 24.8 | 8.32 | 0 | 0.7015 | 0.015 | |
| Kimi-K2.5(think off)Thinking Model Category=Non-Thinking Model, # Tokens=-2026.03 | 31.45 | 72.7 | 77.9 | 21.65 | 5 | 0.285 | 0.052 | |
| Qwen3-30B-A3B-Instruct-2507Thinking Model Category=Non-Thinking Model, # Tokens=-2026.03 | 14.57 | 93.97 | 42.3 | 3.92 | 0 | 0.5625 | 0.0299 |