Commonsense Reasoning on Hellaswag (Score, Relative Change, Gate Density)
78.5ScoreVanilla
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| VanillaAttention Mechanism=Full Attention, Model Size=8.1B, Training Context=32k2026.05 | 78.5 | — | — | |
| Self-Pruned KVAttention Mechanism=Self-Pruned KV, tau=0.5, Model Size=8.1B, Training Context=32k2026.05 | 78.4 | 0 | 40.5 | |
| Olmo3 BaseTuning Protocol=Freezing param., Few-shot settings=10-shot2026.07 | 44.17 | — | — | |
| LMK token tuningTuning Protocol=Freezing param., Few-shot settings=10-shot2026.07 | 43.25 | — | — | |
| Olmo3 512swaTuning Protocol=Full param. CPT, Few-shot settings=10-shot2026.07 | 42.96 | — | — | |
| HiLS-Attn HoPE-Q-CalTuning Protocol=Full param. CPT, Few-shot settings=10-shot2026.07 | 38.71 | — | — | |
| HiLS-Attn NoPE-Q-CalTuning Protocol=Full param. CPT, Few-shot settings=10-shot2026.07 | 34.24 | — | — | |
| HiLS-Attn RoPE-Q-CalTuning Protocol=Full param. CPT, Few-shot settings=10-shot2026.07 | 33.17 | — | — |