Multitask Language Understanding on MMLU (Accuracy and Resource Usage)
69.62AccuracyVanilla
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| VanillaBackbone=Llama3.1-8B2026.04 | 69.62 | 14.82 | 2,883 | 6.9 | |
| CoTBackbone=Qwen2.5-7B2026.04 | 66.5 | 1.77 | 649 | 0.2 | |
| Vanilla + H2OBackbone=Llama3.1-8B2026.04 | 65.92 | 27.11 | 736 | 1.8 | |
| CoTBackbone=Llama3.1-8B2026.04 | 65.82 | 2.39 | 736 | 0.3 | |
| LThinker_thoBackbone=Llama3.1-8B2026.04 | 63.39 | 14.88 | 882 | 1.8 | |
| LThinker_tokBackbone=Llama3.1-8B2026.04 | 61.05 | 15.85 | 1,538 | 3.3 | |
| LThinker_thoBackbone=Qwen2.5-7B2026.04 | 60.47 | 13.09 | 944 | 1.9 | |
| VanillaBackbone=Qwen2.5-7B2026.04 | 59.98 | 20.61 | 3,417 | 10 | |
| Vanilla + H2OBackbone=Qwen2.5-7B2026.04 | 59.69 | 29.02 | 1,024 | 3.2 | |
| AnLLMBackbone=Llama3.1-8B2026.04 | 58.62 | 16.53 | 589 | 1.2 | |
| LThinker_tokBackbone=Qwen2.5-7B2026.04 | 57.35 | 13.8 | 489 | 3.5 | |
| AnLLMBackbone=Qwen2.5-7B2026.04 | 54.63 | 14.13 | 875 | 2 | |
| Distill-R1Backbone=Llama3.1-8B2026.04 | 53.46 | 2.97 | 582 | 0.8 | |
| Distill-R1Backbone=Qwen2.5-7B2026.04 | 51.7 | 14.31 | 2,483 | 7.5 | |
| Vanilla + SepLLMBackbone=Llama3.1-8B2026.04 | 25.12 | 50.11 | 1,024 | 7.5 | |
| Vanilla + SepLLMBackbone=Qwen2.5-7B2026.04 | 10.81 | 53.45 | 1,024 | 9 |