List your company
ResearchFormerly WizWand

Needle-in-a-Haystack

100AccuracyRoPE SSMax
-1.9224.545177.46May 28, 2025Jun 5, 2025Jun 13, 2025Jun 21, 2025Jun 29, 2025Jul 7, 2025Jul 15, 2025
Updated 8mo ago

Evaluation results

MethodLinks
RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=1K2025.05100
RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=1.5K2025.05100
RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=2K2025.05100
RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=3K2025.05100
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=1K2025.05100
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=1.5K2025.05100
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=2K2025.05100
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=3K2025.05100
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=4K2025.05100
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=6K2025.05100
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=8K2025.0598
SpindleKVBackbone=LLaMA3-8b, KV cache reserve ratio=15%2025.0797.9
SpindleKVBackbone=Mistral-7b, KV cache reserve ratio=15%2025.0797.5
PyramidKVBackbone=Mistral-7b, KV cache reserve ratio=15%2025.0796.2
ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=1K2025.0596
KV-Latent Distill (L3-8B)Model=L3-8B, dqk=64, dvo=64, Method=Distill2025.0794
PyramidKVBackbone=LLaMA3-8b, KV cache reserve ratio=15%2025.0793.8
L3-8B (Base)Model=L3-8B, dqk=128, dvo=128, Method=Base2025.0792
KV-Latent Train (L3-8B)Model=L3-8B, dqk=64, dvo=64, Method=Train2025.0792
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=10K2025.0592
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=12K2025.0588
RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=4K2025.0564
PyramidInferBackbone=Mistral-7b, KV cache reserve ratio=15%2025.0762.1
PyramidInferBackbone=LLaMA3-8b, KV cache reserve ratio=15%2025.0761.5
RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=1K2025.0540
L2-7B (Base)Model=L2-7B, dqk=128, dvo=128, Method=Base2025.0732
ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=1.5K2025.0526
KV-Latent Train (L2-7B)Model=L2-7B, dqk=64, dvo=64, Method=Train2025.0724
RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=2K2025.0524
ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=2K2025.0522
RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=1.5K2025.0522
RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=3K2025.0520
ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=3K2025.0518
ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=4K2025.0516
RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=4K2025.0512
ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=6K2025.0510
RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=6K2025.058
KV-Latent Train (L3-8B, d=16)Model=L3-8B, dqk=16, dvo=16, Method=Train2025.076
KV-Latent Distill (L2-7B)Model=L2-7B, dqk=64, dvo=64, Method=Distill2025.074
NoPE SSMaxPosition Encoding=NoPE SSMax, Model Size=120M, Context Length=1K2025.054
ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=10K2025.054
ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=8K2025.052
ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=12K2025.052
RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=10K2025.052