List your company
ResearchFormerly WizWand

Needle-in-the-haystack Retrieval on Needle-in-the-haystack 2500 tokens

100Needle AccuracyBaseline
83.3687.689296.32Apr 23, 2026
Updated 6mo ago

Evaluation results

MethodLinks
BaselineModel=Qwen2.5-14B / 32B / 72B, Tokens kept=100%, V groups kept=100%2026.04100100
Expected AttentionModel=Qwen2.5-14B / 32B / 72B, Tokens kept=62.5%, V groups kept=100%2026.0410062.5
Expected AttentionModel=Qwen2.5-14B / 32B / 72B, Tokens kept=75%, V groups kept=100%2026.0410075
Sub-tokenModel=Qwen2.5-14B / 32B / 72B, Tokens kept=100%, V groups kept=25%2026.0410062.5
Sub-tokenModel=Qwen2.5-14B / 32B / 72B, Tokens kept=100%, V groups kept=50%2026.0410075
EA + Sub-tokenModel=Qwen2.5-14B / 32B / 72B, Tokens kept=75%, V groups kept=75%2026.0410065.6
EA + Sub-tokenModel=Qwen2.5-14B / 32B / 72B, Tokens kept=75%, V groups kept=50%2026.0410056.2
EA + Sub-tokenModel=Qwen2.5-14B / 32B / 72B, Tokens kept=50%, V groups kept=50%2026.0410037.5
BaselineModel=Qwen2.5-7B, Tokens kept=100%, V groups kept=100%2026.0485100
Expected AttentionModel=Qwen2.5-7B, Tokens kept=62.5%, V groups kept=100%2026.048562.5
Expected AttentionModel=Qwen2.5-7B, Tokens kept=75%, V groups kept=100%2026.048575
EA + Sub-tokenModel=Qwen2.5-7B, Tokens kept=75%, V groups kept=75%2026.048565.6
EA + Sub-tokenModel=Qwen2.5-7B, Tokens kept=75%, V groups kept=50%2026.048556.2
EA + Sub-tokenModel=Qwen2.5-7B, Tokens kept=50%, V groups kept=50%2026.0484.537.5
Sub-tokenModel=Qwen2.5-7B, Tokens kept=100%, V groups kept=25%2026.048462.5
Sub-tokenModel=Qwen2.5-7B, Tokens kept=100%, V groups kept=50%2026.048475