Needle-in-a-Haystack
100AccuracyRoPE SSMax
Evaluation Results
| Method | Links | |
|---|---|---|
| RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=1K2025.05 | 100 | |
| RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=1.5K2025.05 | 100 | |
| RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=2K2025.05 | 100 | |
| RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=3K2025.05 | 100 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=1K2025.05 | 100 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=1.5K2025.05 | 100 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=2K2025.05 | 100 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=3K2025.05 | 100 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=4K2025.05 | 100 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=6K2025.05 | 100 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=8K2025.05 | 98 | |
| SpindleKVBackbone=LLaMA3-8b, KV cache reserve ratio=15%2025.07 | 97.9 | |
| SpindleKVBackbone=Mistral-7b, KV cache reserve ratio=15%2025.07 | 97.5 | |
| PyramidKVBackbone=Mistral-7b, KV cache reserve ratio=15%2025.07 | 96.2 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=1K2025.05 | 96 | |
| KV-Latent Distill (L3-8B)Model=L3-8B, dqk=64, dvo=64, Method=Distill2025.07 | 94 | |
| PyramidKVBackbone=LLaMA3-8b, KV cache reserve ratio=15%2025.07 | 93.8 | |
| L3-8B (Base)Model=L3-8B, dqk=128, dvo=128, Method=Base2025.07 | 92 | |
| KV-Latent Train (L3-8B)Model=L3-8B, dqk=64, dvo=64, Method=Train2025.07 | 92 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=10K2025.05 | 92 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=12K2025.05 | 88 | |
| RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=4K2025.05 | 64 | |
| PyramidInferBackbone=Mistral-7b, KV cache reserve ratio=15%2025.07 | 62.1 | |
| PyramidInferBackbone=LLaMA3-8b, KV cache reserve ratio=15%2025.07 | 61.5 | |
| RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=1K2025.05 | 40 | |
| L2-7B (Base)Model=L2-7B, dqk=128, dvo=128, Method=Base2025.07 | 32 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=1.5K2025.05 | 26 | |
| KV-Latent Train (L2-7B)Model=L2-7B, dqk=64, dvo=64, Method=Train2025.07 | 24 | |
| RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=2K2025.05 | 24 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=2K2025.05 | 22 | |
| RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=1.5K2025.05 | 22 | |
| RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=3K2025.05 | 20 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=3K2025.05 | 18 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=4K2025.05 | 16 | |
| RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=4K2025.05 | 12 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=6K2025.05 | 10 | |
| RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=6K2025.05 | 8 | |
| KV-Latent Train (L3-8B, d=16)Model=L3-8B, dqk=16, dvo=16, Method=Train2025.07 | 6 | |
| KV-Latent Distill (L2-7B)Model=L2-7B, dqk=64, dvo=64, Method=Distill2025.07 | 4 | |
| NoPE SSMaxPosition Encoding=NoPE SSMax, Model Size=120M, Context Length=1K2025.05 | 4 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=10K2025.05 | 4 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=8K2025.05 | 2 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=12K2025.05 | 2 | |
| RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=10K2025.05 | 2 |