Big-Bench Hard Reasoning on BBH
69.16AccuracyTokenSkip
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| TokenSkipModel Series=Llama3.1-8B, Serving Setting=Throughput2026.04 | 69.16 | 34.2 | 2,581 | 7.6 | |
| LThinker++Model Series=Llama3.1-8B, Serving Setting=Throughput2026.04 | 69.09 | 21.9 | 896 | 1.7 | |
| VanillaModel Series=Llama3.1-8B, Serving Setting=Throughput2026.04 | 67.68 | 28.5 | 2,826 | 8.8 | |
| CoTModel Series=Llama3.1-8B, Serving Setting=Throughput2026.04 | 66.46 | 55.8 | 642 | 0.2 | |
| CoTModel Series=Qwen2.5-7B, Serving Setting=Throughput2026.04 | 65.45 | 40.8 | 570 | 0.1 | |
| VanillaModel Series=Qwen2.5-7B, Serving Setting=Throughput2026.04 | 61.55 | 24.7 | 3,205 | 10.2 | |
| Distill-R1Model Series=Llama3.1-8B, Serving Setting=Throughput2026.04 | 61.21 | 49.8 | 380 | 0.2 | |
| LThinker++Model Series=Qwen2.5-7B, Serving Setting=Throughput2026.04 | 61.01 | 28.5 | 927 | 2.5 | |
| TokenSkipModel Series=Qwen2.5-7B, Serving Setting=Throughput2026.04 | 59.87 | 33.2 | 2,850 | 9.3 | |
| LThinker*Model Series=Llama3.1-8B, Serving Setting=Throughput2026.04 | 58.79 | 21.2 | 734 | 1.1 | |
| Distill-R1Model Series=Qwen2.5-7B, Serving Setting=Throughput2026.04 | 57.78 | 331.8 | 1,967 | 6 | |
| LThinker*Model Series=Qwen2.5-7B, Serving Setting=Throughput2026.04 | 51.85 | 36.5 | 851 | 2.9 | |
| MoE + L_ERCNumber of Parameters=15B, Activated Parameters=700M, Loss Function=ERC loss2025.12 | 45.6 | — | — | — | |
| MoENumber of Parameters=15B, Activated Parameters=700M2025.12 | 44.3 | — | — | — |