Inference Efficiency on DeepScaleR-40k (1,024 mathematical problems)
760.74Throughput (tokens/s)G-KV
Evaluation Results
| Method | Links | |
|---|---|---|
| G-KVModel=DeepSeek-Qwen-Distill-7B, Batch Size=128, KV Cache Budget=5122025.11 | 760.74 | |
| G-KVModel=DeepSeek-Llama-Distill-8B, Batch Size=64, KV Cache Budget=5122025.11 | 612.6 | |
| G-KVModel=DeepSeek-Llama-Distill-8B, Batch Size=32, KV Cache Budget=5122025.11 | 517.3 | |
| G-KVModel=DeepSeek-Qwen-Distill-7B, Batch Size=64, KV Cache Budget=5122025.11 | 475.59 | |
| G-KVModel=DeepSeek-Qwen-Distill-7B, Batch Size=128, KV Cache Budget=10242025.11 | 448.35 | |
| G-KVModel=DeepSeek-Qwen-Distill-7B, Batch Size=64, KV Cache Budget=10242025.11 | 367.96 | |
| G-KVModel=DeepSeek-Qwen-Distill-7B, Batch Size=32, KV Cache Budget=5122025.11 | 261.32 | |
| G-KVModel=DeepSeek-Llama-Distill-8B, Batch Size=64, KV Cache Budget=10242025.11 | 258.18 | |
| G-KVModel=DeepSeek-Qwen-Distill-7B, Batch Size=128, KV Cache Budget=20482025.11 | 248.22 | |
| R-KVModel=DeepSeek-Qwen-Distill-7B, Batch Size=128, KV Cache Budget=20482025.11 | 238.43 | |
| G-KVModel=DeepSeek-Qwen-Distill-7B, Batch Size=64, KV Cache Budget=20482025.11 | 221.23 | |
| G-KVModel=DeepSeek-Qwen-Distill-7B, Batch Size=32, KV Cache Budget=10242025.11 | 212.93 | |
| R-KVModel=DeepSeek-Qwen-Distill-7B, Batch Size=64, KV Cache Budget=20482025.11 | 203.66 | |
| G-KVModel=DeepSeek-Llama-Distill-8B, Batch Size=32, KV Cache Budget=10242025.11 | 193.56 | |
| R-KVModel=DeepSeek-Qwen-Distill-7B, Batch Size=32, KV Cache Budget=20482025.11 | 172.44 | |
| G-KVModel=DeepSeek-Qwen-Distill-7B, Batch Size=32, KV Cache Budget=20482025.11 | 170.64 | |
| G-KVModel=DeepSeek-Llama-Distill-8B, Batch Size=16, KV Cache Budget=5122025.11 | 158.43 | |
| G-KVModel=DeepSeek-Llama-Distill-8B, Batch Size=64, KV Cache Budget=20482025.11 | 154.52 | |
| G-KVModel=DeepSeek-Llama-Distill-8B, Batch Size=32, KV Cache Budget=20482025.11 | 118.82 | |
| G-KVModel=DeepSeek-Llama-Distill-8B, Batch Size=16, KV Cache Budget=10242025.11 | 118.1 | |
| R-KVModel=DeepSeek-Llama-Distill-8B, Batch Size=64, KV Cache Budget=20482025.11 | 111.89 | |
| R-KVModel=DeepSeek-Llama-Distill-8B, Batch Size=32, KV Cache Budget=20482025.11 | 98.01 | |
| G-KVModel=DeepSeek-Llama-Distill-8B, Batch Size=16, KV Cache Budget=20482025.11 | 93.91 | |
| R-KVModel=DeepSeek-Llama-Distill-8B, Batch Size=16, KV Cache Budget=20482025.11 | 82.33 | |
| Full-KVModel=DeepSeek-Qwen-Distill-7B, Batch Size=322025.11 | 62.41 | |
| Full-KVModel=DeepSeek-Llama-Distill-8B, Batch Size=162025.11 | 31.08 |