List your company
ResearchFormerly WizWand

In-context Retrieval on RULER 8K

100S-NFull-Attn RoPE
14.7236.865981.14Jul 3, 2026
Updated 3mo ago

Evaluation results

MethodLinks
Full-Attn RoPEParameters=345M, Training context length=8K, Extra parameters=None2026.071009734
Full-Attn HoPEParameters=345M, Training context length=8K, Extra parameters=None2026.071009836
Full-Attn HoPE (extra MLP)Parameters=345M, Training context length=8K, Extra parameters=0.6%2026.071009939
Naive-BSA-HoPEParameters=345M, Training context length=8K, Extra parameters=None2026.071009723
HiLS-Attn-HoPEParameters=345M, Training context length=8K, Extra parameters=0.6%2026.071009572
Dash-Attention-RoPEParameters=345M, Training context length=8K, Extra parameters=0.006%2026.07926734
HSA-UltralongParameters=345M, Training context length=8K, Extra parameters=12.7%2026.07877923
InfLLM v2*Parameters=345M, Training context length=8K, Extra parameters=0.006%2026.07773925
NSA-RoPEParameters=345M, Training context length=8K, Extra parameters=0.2%2026.07491716
SWA-RoPEParameters=345M, Training context length=8K, Extra parameters=None2026.0718822