Multi-task Language Understanding on LM Evaluation Harness (test)
44.28ARC Challenge AccGQA
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GQAModel Scale=1.5B, Evaluation Protocol=5-shot2025.12 | 44.28 | 73.91 | — | — | 60.16 | 41.2 | — | — | 54.58 | 34.63 | 92.4 | 35.49 | |
| FusedKVModel Scale=1.5B, Evaluation Protocol=5-shot2025.12 | 44.2 | 74.2 | — | — | 61.88 | 43.33 | — | — | 55.82 | 37.43 | 93.5 | 36.18 | |
| FusedKV-LiteModel Scale=1.5B, Evaluation Protocol=5-shot2025.12 | 43.52 | 74.79 | — | — | 60.77 | 41.61 | — | — | 55.3 | 36.43 | 93.7 | 36.29 | |
| YOCOModel Scale=1.5B, Evaluation Protocol=5-shot2025.12 | 42.92 | 73.19 | — | — | 60.09 | 40.81 | — | — | 54.19 | 35.3 | 91.7 | 35.35 | |
| VanillaModel Scale=1.5B, Cache Mem.=1, Evaluation Protocol=5-shot2025.12 | 42.58 | 72.55 | — | — | 60.67 | 40.54 | — | — | 54.55 | 35.69 | 94.7 | 35.12 | |
| CLAModel Scale=1.5B, Cache Mem.=1/2, Evaluation Protocol=5-shot2025.12 | 42.32 | 73.99 | — | — | 59.04 | 39.61 | — | — | 53.91 | 35 | 92.6 | 34.83 | |
| FusedKV-LiteModel Scale=650M, Evaluation Protocol=5-shot2025.12 | 37.46 | 66.58 | — | — | 49.53 | 32.95 | — | — | 48.51 | 34.24 | 86.9 | 31.88 | |
| VanillaModel Scale=650M, Cache Mem.=1, Evaluation Protocol=5-shot2025.12 | 36.26 | 65.36 | — | — | 49.13 | 31.67 | — | — | 48.08 | 34.39 | 88.1 | 31.62 | |
| UniformNumber of Training Tokens=1 Billion2023.07 | 35.4 | 62.2 | 68.9 | 81 | 63.9 | 64.4 | 74.8 | 62.8 | 64.2 | — | — | — | |
| UniformNumber of Training Tokens=2 Billion2023.07 | 35.3 | 62.4 | 67.7 | 80 | 63.8 | 65.9 | 75.5 | 63.9 | 64.3 | — | — | — | |
| SKILL-ITNumber of Training Tokens=2 Billion2023.07 | 34.9 | 61.7 | 68.6 | 81 | 63.9 | 66.7 | 75.2 | 63.2 | 64.4 | — | — | — | |
| SKILL-ITNumber of Training Tokens=3 Billion2023.07 | 34.8 | 62 | 68.7 | 81 | 63.9 | 66 | 75.7 | 63.1 | 64.4 | — | — | — | |
| SKILL-ITNumber of Training Tokens=1 Billion2023.07 | 34.6 | 61.2 | 68.2 | 82 | 63.7 | 67 | 75 | 63.9 | 64.4 | — | — | — | |
| UniformNumber of Training Tokens=3 Billion2023.07 | 34.6 | 62.5 | 67.2 | 81 | 64 | 66.8 | 75 | 63.4 | 64.3 | — | — | — | |
| YOCOModel Scale=650M, Evaluation Protocol=5-shot2025.12 | 34.39 | 64.14 | — | — | 48.65 | 30.74 | — | — | 47.18 | 34.96 | 86.7 | 30.66 | |
| FusedKVModel Scale=650M, Evaluation Protocol=5-shot2025.12 | 34.39 | 65.61 | — | — | 49.68 | 31.03 | — | — | 47.45 | 33.33 | 86.6 | 31.54 | |
| GQAModel Scale=650M, Evaluation Protocol=5-shot2025.12 | 34.04 | 65.36 | — | — | 48.62 | 30.56 | — | — | 47.08 | 33.11 | 86.3 | 31.56 | |
| CLAModel Scale=650M, Cache Mem.=1/2, Evaluation Protocol=5-shot2025.12 | 32.94 | 65.91 | — | — | 47.48 | 27.94 | — | — | 46.49 | 34.25 | 86.1 | 30.81 | |
| FusedKV-LiteModel Scale=332M, Evaluation Protocol=5-shot2025.12 | 32 | 58.67 | — | — | 42.07 | 26.39 | — | — | 43.19 | 34.22 | 79.6 | 29.38 | |
| GQAModel Scale=332M, Evaluation Protocol=5-shot2025.12 | 31.4 | 60.23 | — | — | 41.72 | 25.52 | — | — | 43.4 | 34.99 | 80.5 | 29.45 | |
| YOCOModel Scale=332M, Evaluation Protocol=5-shot2025.12 | 30.29 | 59.93 | — | — | 41.74 | 26.61 | — | — | 43.08 | 33.64 | 80.6 | 28.77 | |
| FusedKVModel Scale=332M, Evaluation Protocol=5-shot2025.12 | 30.12 | 60.98 | — | — | 42.74 | 28.1 | — | — | 43.59 | 33.62 | 80.3 | 29.26 | |
| VanillaModel Scale=332M, Cache Mem.=1, Evaluation Protocol=5-shot2025.12 | 29.69 | 59.26 | — | — | 42.12 | 24.49 | — | — | 43.33 | 34.64 | 84.2 | 28.93 | |
| CLAModel Scale=332M, Cache Mem.=1/2, Evaluation Protocol=5-shot2025.12 | 27.65 | 54.67 | — | — | 40.75 | 13.51 | — | — | 39.52 | 34.8 | 76.7 | 28.56 |