Question Answering on OpenBookQA (Score, Relative Change, Avg. Gate Density)
45OpenBookQA ScoreVanilla
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| VanillaAttention Mechanism=Full Attention, Model Size=8.1B, Training Context=32k2026.05 | 45 | — | — | |
| Self-Pruned KVAttention Mechanism=Self-Pruned KV, tau=0.5, Model Size=8.1B, Training Context=32k2026.05 | 43 | -4.4 | 49.9 |