Needle In A Haystack on Ruler NIAH (Single 2)
1AccuracyALiBi SSMax
Evaluation Results
| Method | Links | |
|---|---|---|
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=1K2025.05 | 1 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=1K2025.05 | 1 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=1.5K2025.05 | 1 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=2K2025.05 | 1 | |
| RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=1K2025.05 | 0.98 | |
| RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=1.5K2025.05 | 0.96 | |
| RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=2K2025.05 | 0.94 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=3K2025.05 | 0.88 | |
| RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=3K2025.05 | 0.84 | |
| RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=1K2025.05 | 0.84 | |
| NoPE SSMaxPosition Encoding=NoPE SSMax, Model Size=120M, Context Length=1K2025.05 | 0.7 | |
| Sinusoidal SSMaxPosition Encoding=Sinusoidal SSMax, Model Size=120M, Context Length=1K2025.05 | 0.7 | |
| RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=4K2025.05 | 0.52 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=1.5K2025.05 | 0.46 | |
| RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=1.5K2025.05 | 0.36 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=4K2025.05 | 0.24 | |
| RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=3K2025.05 | 0.14 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=2K2025.05 | 0.12 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=3K2025.05 | 0.1 | |
| RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=2K2025.05 | 0.1 | |
| RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=4K2025.05 | 0.1 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=4K2025.05 | 0.06 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=6K2025.05 | 0.06 | |
| ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=6K2025.05 | 0.02 | |
| BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=8K2025.05 | 0.02 |