Commonsense Reasoning on Winogrande (Score, Relative Change, Avg. Gate Density)
71.7ScoreSelf-Pruned KV
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Self-Pruned KVAttention Mechanism=Self-Pruned KV, tau=0.5, Model Size=8.1B, Training Context=32k2026.05 | 71.7 | 0.8 | 46.5 | |
| VanillaAttention Mechanism=Full Attention, Model Size=8.1B, Training Context=32k2026.05 | 71.1 | — | — |