List your company
ResearchFormerly WizWand

Long-Context Retrieval on RULER 16K

98.67ScoreHiLS-Attn HoPE-Q-Cal
-1.866824.234150.33576.4359Apr 27, 2026May 8, 2026May 19, 2026May 30, 2026Jun 10, 2026Jun 21, 2026Jul 3, 2026
Updated 3mo ago

Evaluation results

MethodLinks
HiLS-Attn HoPE-Q-CalTuning Protocol=Full param. CPT, Positional Encoding=HoPE-Q-Cal2026.0798.67
Ministral-3-8B-Instruct-2512-BF16max context=256k2026.0593.4
Llama-3.1-8B-Instructmax context=128k2026.0593.3
Qwen3-8Bmax context=32k2026.0592
GPT-5 nanomax context=400k2026.0591.7
Qwen3-4Bmax context=32k2026.0591
Full AttentionSetting=CPT setting2026.0489.12
Hybrid-KSASetting=CPT setting2026.0488.86
gemma-3-12b-itmax context=128k2026.0587.5
KSASetting=CPT setting2026.0484.78
Llama-3.2-3B-Instructmax context=128k2026.0583.1
Hybrid-SWASetting=CPT setting2026.0482.87
Hybrid-SCASetting=CPT setting2026.0480.67
Hybrid-LinearSetting=CPT setting2026.0478.06
gpt-oss-20bmax context=128k2026.0577.2
Hybrid-KSAsetting=Train-from-scratch2026.0474.07
Hybrid-GDNsetting=Train-from-scratch2026.0474.04
HiLS-Attn NoPE-Q-CalTuning Protocol=Full param. CPT, Positional Encoding=NoPE-Q-Cal2026.0773.33
Fullsetting=Train-from-scratch2026.0473.24
Hybrid-SCAsetting=Train-from-scratch2026.0472.55
Hybrid-SWAsetting=Train-from-scratch2026.0469.54
gemma-3-4b-itmax context=128k2026.0568.5
KSAsetting=Train-from-scratch2026.0466.74
FastwebMIIA-7Bmax context=16k2026.0558.7
EngGPT2-16B-A3Bmax context=32k2026.0557.9
Moonlight-16B-A3B-Instructmax context=128k2026.0555.7
HiLS-Attn RoPE-Q-CalTuning Protocol=Full param. CPT, Positional Encoding=RoPE-Q-Cal2026.0750.67
LLaMAntino-3-ANITA-8B-Inst-DPO-ITAmax context=8k2026.0548.2
Velvet-14Bmax context=128k2026.0534.4
Olmo3 512swaTuning Protocol=Full param. CPT2026.0733
deepseek-moe-16b-chatmax context=4k2026.0525.2
Minerva-7B-instruct-v1,0max context=4k2026.0516.1
Olmo3 BaseTuning Protocol=Freezing param.2026.073.67
LMK token tuningTuning Protocol=Freezing param.2026.072