Needle In A Haystack on Ruler NIAH Single 3
84AccuracyBAM SSMax
Evaluation Results
| Method | Links | |
|---|---|---|
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=1K2025.05 | 84 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=1.5K2025.05 | 68 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=2K2025.05 | 42 | |
| RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=1.5K2025.05 | 30 | |
| RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=1K2025.05 | 28 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=1K2025.05 | 24 | |
| RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=2K2025.05 | 10 | |
| RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=1K2025.05 | 10 | |
| Sinusoidal SSMaxPosition Encoding=Sinusoidal SSMax, Model Size=120M, Context Length=1K2025.05 | 8 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=3K2025.05 | 8 | |
| RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=3K2025.05 | 6 | |
| NoPE SSMaxPosition Encoding=NoPE SSMax, Model Size=120M, Context Length=1K2025.05 | 4 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=1.5K2025.05 | 4 |