Question Answering on Natural Questions (NQ) (Score, Relative Change, and Gate Density Metrics)
0.227ScoreSelf-Pruned KV
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Self-Pruned KVAttention Mechanism=Self-Pruned KV, tau=0.5, Model Size=8.1B, Training Context=32k2026.05 | 0.227 | 1.7 | 29.5 | |
| VanillaAttention Mechanism=Full Attention, Model Size=8.1B, Training Context=32k2026.05 | 0.223 | — | — |