Needle-in-the-haystack Retrieval on Needle-in-the-haystack 2500 tokens
100Needle AccuracyBaseline
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| BaselineModel=Qwen2.5-14B / 32B / 72B, Tokens kept=100%, V groups kept=100%2026.04 | 100 | 100 | |
| Expected AttentionModel=Qwen2.5-14B / 32B / 72B, Tokens kept=62.5%, V groups kept=100%2026.04 | 100 | 62.5 | |
| Expected AttentionModel=Qwen2.5-14B / 32B / 72B, Tokens kept=75%, V groups kept=100%2026.04 | 100 | 75 | |
| Sub-tokenModel=Qwen2.5-14B / 32B / 72B, Tokens kept=100%, V groups kept=25%2026.04 | 100 | 62.5 | |
| Sub-tokenModel=Qwen2.5-14B / 32B / 72B, Tokens kept=100%, V groups kept=50%2026.04 | 100 | 75 | |
| EA + Sub-tokenModel=Qwen2.5-14B / 32B / 72B, Tokens kept=75%, V groups kept=75%2026.04 | 100 | 65.6 | |
| EA + Sub-tokenModel=Qwen2.5-14B / 32B / 72B, Tokens kept=75%, V groups kept=50%2026.04 | 100 | 56.2 | |
| EA + Sub-tokenModel=Qwen2.5-14B / 32B / 72B, Tokens kept=50%, V groups kept=50%2026.04 | 100 | 37.5 | |
| BaselineModel=Qwen2.5-7B, Tokens kept=100%, V groups kept=100%2026.04 | 85 | 100 | |
| Expected AttentionModel=Qwen2.5-7B, Tokens kept=62.5%, V groups kept=100%2026.04 | 85 | 62.5 | |
| Expected AttentionModel=Qwen2.5-7B, Tokens kept=75%, V groups kept=100%2026.04 | 85 | 75 | |
| EA + Sub-tokenModel=Qwen2.5-7B, Tokens kept=75%, V groups kept=75%2026.04 | 85 | 65.6 | |
| EA + Sub-tokenModel=Qwen2.5-7B, Tokens kept=75%, V groups kept=50%2026.04 | 85 | 56.2 | |
| EA + Sub-tokenModel=Qwen2.5-7B, Tokens kept=50%, V groups kept=50%2026.04 | 84.5 | 37.5 | |
| Sub-tokenModel=Qwen2.5-7B, Tokens kept=100%, V groups kept=25%2026.04 | 84 | 62.5 | |
| Sub-tokenModel=Qwen2.5-7B, Tokens kept=100%, V groups kept=50%2026.04 | 84 | 75 |