Long-context Variable Tracking on variable-tracking
79.5AccuracyQwen2.5-72B Baseline
Evaluation Results
| Method | Links | |
|---|---|---|
| Qwen2.5-72B BaselineModel scale=72B, Compression Method=Baseline, KV retention=100%2026.04 | 79.5 | |
| Sub-token routing (Qwen2.5-72B)Model scale=72B, Compression Method=Sub-token routing, KV retention=62.5%2026.04 | 79.5 | |
| Expected Attention (Qwen2.5-72B)Model scale=72B, Compression Method=EA, KV retention=62.5%2026.04 | 79.5 | |
| EA + Sub-token routing (Qwen2.5-72B)Model scale=72B, Compression Method=EA+Sub-token, KV retention=37.5%2026.04 | 79.5 | |
| Qwen2.5-32B BaselineModel scale=32B, Compression Method=Baseline, KV retention=100%2026.04 | 79 | |
| Sub-token routing (Qwen2.5-32B)Model scale=32B, Compression Method=Sub-token routing, KV retention=62.5%2026.04 | 79 | |
| Expected Attention (Qwen2.5-32B)Model scale=32B, Compression Method=EA, KV retention=62.5%2026.04 | 79 | |
| EA + Sub-token routing (Qwen2.5-32B)Model scale=32B, Compression Method=EA+Sub-token, KV retention=37.5%2026.04 | 79 | |
| Qwen2.5-14B BaselineModel scale=14B, Compression Method=Baseline, KV retention=100%2026.04 | 77 | |
| Sub-token routing (Qwen2.5-14B)Model scale=14B, Compression Method=Sub-token routing, KV retention=62.5%2026.04 | 77 | |
| Expected Attention (Qwen2.5-14B)Model scale=14B, Compression Method=EA, KV retention=62.5%2026.04 | 77 | |
| EA + Sub-token routing (Qwen2.5-14B)Model scale=14B, Compression Method=EA+Sub-token, KV retention=37.5%2026.04 | 77 | |
| Qwen2.5-7B BaselineModel scale=7B, Compression Method=Baseline, KV retention=100%2026.04 | 29.5 | |
| Sub-token routing (Qwen2.5-7B)Model scale=7B, Compression Method=Sub-token routing, KV retention=62.5%2026.04 | 29.5 | |
| Expected Attention (Qwen2.5-7B)Model scale=7B, Compression Method=EA, KV retention=62.5%2026.04 | 29.5 | |
| EA + Sub-token routing (Qwen2.5-7B)Model scale=7B, Compression Method=EA+Sub-token, KV retention=37.5%2026.04 | 29.5 |