Complex Reasoning and Retrieval on LMB, MMLU, GSM8K, SWDE, KV-Ret, COV
60.1LMB AccuracyLLAMA-3.2-1B
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| LLAMA-3.2-1Bnumber of attention heads=5122026.02 | 60.1 | 46 | 33.1 | 78.6 | 99.3 | 100 | |
| Hybrid-Qwen2-1.5Bnumber of attention heads=3362026.02 | 60.1 | 59.8 | 42.6 | 81.4 | 100 | 99.3 | |
| QWEN2-1.5Bnumber of attention heads=3362026.02 | 60 | 60.1 | 43.8 | 82.5 | 100 | 100 | |
| Hybrid-Qwen2-1.5Bnumber of attention heads=302026.02 | 58 | 57 | 43.1 | 81.6 | 100 | 98.1 | |
| Hybrid-Qwen2-1.5Bnumber of attention heads=202026.02 | 57.5 | 56.3 | 40.5 | 81.3 | 100 | 96.9 | |
| Hybrid-Qwen2-1.5Bnumber of attention heads=102026.02 | 57.1 | 55.4 | 40.5 | 81 | 100 | 96.4 | |
| Hybrid-Qwen2-1.5Bnumber of attention heads=52026.02 | 56.6 | 53.2 | 35.3 | 78 | 90 | 90.4 | |
| Hybrid-Llama-1Bnumber of attention heads=5122026.02 | 56 | 45 | 35.8 | 75.3 | 99.4 | 98.2 | |
| Hybrid-Llama-1Bnumber of attention heads=202026.02 | 55 | 43 | 34 | 72.5 | 99.3 | 95.8 | |
| Hybrid-Llama-1Bnumber of attention heads=102026.02 | 54.2 | 42.1 | 34.4 | 71.1 | 99 | 95 | |
| Hybrid-Llama-1Bnumber of attention heads=302026.02 | 54 | 43.4 | 33.1 | 70.4 | 98 | 94.3 | |
| Hybrid-Llama-1Bnumber of attention heads=402026.02 | 54 | 44 | 34 | 71.1 | 99.4 | 95.4 | |
| Hybrid-Llama-1Bnumber of attention heads=52026.02 | 53.6 | 40.3 | 30 | 66 | 90 | 88.3 | |
| Hybrid-Qwen2-1.5Bnumber of attention heads=02026.02 | 53.6 | 51.9 | 26.8 | 32.1 | 16 | 52.1 | |
| Hybrid-Llama-1Bnumber of attention heads=02026.02 | 50.7 | 39.2 | 25.1 | 27.7 | 13.2 | 49.2 |