Multi-task Language Understanding on MMLU (Accuracy, Latency, Memory)
66.5AccuracyCoT
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| CoTModel Series=Qwen2.5-7B, Serving Setting=Throughput2026.04 | 66.5 | 106.2 | 649 | 0.2 | |
| CoTModel Series=Llama3.1-8B, Serving Setting=Throughput2026.04 | 65.82 | 143.4 | 736 | 0.3 | |
| LThinker++Model Series=Llama3.1-8B, Serving Setting=Throughput2026.04 | 61.77 | 31.4 | 883 | 1.4 | |
| VanillaModel Series=Llama3.1-8B, Serving Setting=Throughput2026.04 | 61.15 | 45.8 | 2,570 | 7.1 | |
| TokenSkipModel Series=Llama3.1-8B, Serving Setting=Throughput2026.04 | 57.06 | 77.9 | 2,499 | 6.9 | |
| LThinker*Model Series=Llama3.1-8B, Serving Setting=Throughput2026.04 | 56.8 | 35.2 | 782 | 1.3 | |
| VanillaModel Series=Qwen2.5-7B, Serving Setting=Throughput2026.04 | 55.27 | 33.1 | 2,682 | 7.7 | |
| LThinker++Model Series=Qwen2.5-7B, Serving Setting=Throughput2026.04 | 55.05 | 31.8 | 755 | 1.6 | |
| TokenSkipModel Series=Qwen2.5-7B, Serving Setting=Throughput2026.04 | 54.4 | 72.3 | 2,831 | 9.1 | |
| Distill-R1Model Series=Llama3.1-8B, Serving Setting=Throughput2026.04 | 53.46 | 178.2 | 582 | 0.8 | |
| LThinker*Model Series=Qwen2.5-7B, Serving Setting=Throughput2026.04 | 52.87 | 41.7 | 718 | 1.6 | |
| Distill-R1Model Series=Qwen2.5-7B, Serving Setting=Throughput2026.04 | 51.7 | 858.6 | 2,483 | 7.5 |