Natural Language Understanding on ARC-C, ARC-E, PIQA, WG, HS, OBQA
48.2ARC-C AccuracyHybrid-Qwen2-1.5B
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Hybrid-Qwen2-1.5Bnumber of attention heads=3362026.02 | 48.2 | 77.6 | 75.9 | 63.6 | 67.5 | 40 | 100.5 | |
| Hybrid-Qwen2-1.5Bnumber of attention heads=52026.02 | 47.1 | 76 | 75.8 | 62.9 | 65.7 | 40.2 | 99.2 | |
| Hybrid-Qwen2-1.5Bnumber of attention heads=102026.02 | 47.1 | 75.8 | 75.6 | 63 | 65.3 | 40.4 | 99 | |
| Hybrid-Qwen2-1.5Bnumber of attention heads=02026.02 | 46.5 | 76.3 | 75.5 | 64 | 65.1 | 40 | 99.1 | |
| QWEN2-1.5Bnumber of attention heads=3362026.02 | 46.4 | 76.6 | 75.9 | 63.3 | 68.2 | 40.4 | 100 | |
| Hybrid-Qwen2-1.5Bnumber of attention heads=302026.02 | 45.9 | 76.4 | 76.1 | 63.4 | 66.1 | 41.2 | 99.5 | |
| Hybrid-Qwen2-1.5Bnumber of attention heads=202026.02 | 45 | 76.3 | 76 | 62.4 | 65.9 | 41.2 | 98.9 | |
| Hybrid-Llama-1Bnumber of attention heads=302026.02 | 39.3 | 69.3 | 75 | 61.5 | 62.2 | 38.4 | 102.9 | |
| Hybrid-Llama-1Bnumber of attention heads=202026.02 | 38.2 | 69.3 | 74.5 | 62.9 | 61.1 | 36.5 | 101.9 | |
| LLAMA-3.2-1Bnumber of attention heads=5122026.02 | 38.1 | 68.5 | 74.4 | 59.7 | 60.8 | 34.6 | 100 | |
| Hybrid-Llama-1Bnumber of attention heads=02026.02 | 38 | 69.3 | 74.2 | 61.7 | 61 | 36.6 | 101.4 | |
| Hybrid-Llama-1Bnumber of attention heads=5122026.02 | 37.8 | 70.1 | 74.4 | 61.7 | 62.7 | 39.4 | 103 | |
| Hybrid-Llama-1Bnumber of attention heads=102026.02 | 37.6 | 69 | 74.6 | 60.5 | 62 | 36.8 | 101.3 | |
| Hybrid-Llama-1Bnumber of attention heads=402026.02 | 37.5 | 68.9 | 73.7 | 61.8 | 59.2 | 37.6 | 100.8 | |
| Hybrid-Llama-1Bnumber of attention heads=52026.02 | 36.8 | 69.4 | 73.7 | 60.8 | 58.8 | 36.8 | 100.1 |