List your company
ResearchFormerly WizWand

Needle In A Haystack on Ruler NIAH (Single 2)

1AccuracyALiBi SSMax
-0.01920.24540.510.7746May 28, 2025
Updated 8mo ago

Evaluation results

MethodLinks
ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=1K2025.051
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=1K2025.051
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=1.5K2025.051
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=2K2025.051
RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=1K2025.050.98
RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=1.5K2025.050.96
RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=2K2025.050.94
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=3K2025.050.88
RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=3K2025.050.84
RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=1K2025.050.84
NoPE SSMaxPosition Encoding=NoPE SSMax, Model Size=120M, Context Length=1K2025.050.7
Sinusoidal SSMaxPosition Encoding=Sinusoidal SSMax, Model Size=120M, Context Length=1K2025.050.7
RoPE SSMaxPosition Encoding=RoPE SSMax, Model Size=120M, Context Length=4K2025.050.52
ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=1.5K2025.050.46
RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=1.5K2025.050.36
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=4K2025.050.24
RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=3K2025.050.14
ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=2K2025.050.12
ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=3K2025.050.1
RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=2K2025.050.1
RoPE LocalPosition Encoding=RoPE Local, Model Size=120M, Context Length=4K2025.050.1
ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=4K2025.050.06
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=6K2025.050.06
ALiBi SSMaxPosition Encoding=ALiBi SSMax, Model Size=120M, Context Length=6K2025.050.02
BAM SSMaxPosition Encoding=BAM SSMax, Model Size=120M, Context Length=8K2025.050.02