Long-Context Retrieval on RULER 16K
98.67ScoreHiLS-Attn HoPE-Q-Cal
Evaluation Results
| Method | Links | |
|---|---|---|
| HiLS-Attn HoPE-Q-CalTuning Protocol=Full param. CPT, Positional Encoding=HoPE-Q-Cal2026.07 | 98.67 | |
| Ministral-3-8B-Instruct-2512-BF16max context=256k2026.05 | 93.4 | |
| Llama-3.1-8B-Instructmax context=128k2026.05 | 93.3 | |
| Qwen3-8Bmax context=32k2026.05 | 92 | |
| GPT-5 nanomax context=400k2026.05 | 91.7 | |
| Qwen3-4Bmax context=32k2026.05 | 91 | |
| Full AttentionSetting=CPT setting2026.04 | 89.12 | |
| Hybrid-KSASetting=CPT setting2026.04 | 88.86 | |
| gemma-3-12b-itmax context=128k2026.05 | 87.5 | |
| KSASetting=CPT setting2026.04 | 84.78 | |
| Llama-3.2-3B-Instructmax context=128k2026.05 | 83.1 | |
| Hybrid-SWASetting=CPT setting2026.04 | 82.87 | |
| Hybrid-SCASetting=CPT setting2026.04 | 80.67 | |
| Hybrid-LinearSetting=CPT setting2026.04 | 78.06 | |
| gpt-oss-20bmax context=128k2026.05 | 77.2 | |
| Hybrid-KSAsetting=Train-from-scratch2026.04 | 74.07 | |
| Hybrid-GDNsetting=Train-from-scratch2026.04 | 74.04 | |
| HiLS-Attn NoPE-Q-CalTuning Protocol=Full param. CPT, Positional Encoding=NoPE-Q-Cal2026.07 | 73.33 | |
| Fullsetting=Train-from-scratch2026.04 | 73.24 | |
| Hybrid-SCAsetting=Train-from-scratch2026.04 | 72.55 | |
| Hybrid-SWAsetting=Train-from-scratch2026.04 | 69.54 | |
| gemma-3-4b-itmax context=128k2026.05 | 68.5 | |
| KSAsetting=Train-from-scratch2026.04 | 66.74 | |
| FastwebMIIA-7Bmax context=16k2026.05 | 58.7 | |
| EngGPT2-16B-A3Bmax context=32k2026.05 | 57.9 | |
| Moonlight-16B-A3B-Instructmax context=128k2026.05 | 55.7 | |
| HiLS-Attn RoPE-Q-CalTuning Protocol=Full param. CPT, Positional Encoding=RoPE-Q-Cal2026.07 | 50.67 | |
| LLaMAntino-3-ANITA-8B-Inst-DPO-ITAmax context=8k2026.05 | 48.2 | |
| Velvet-14Bmax context=128k2026.05 | 34.4 | |
| Olmo3 512swaTuning Protocol=Full param. CPT2026.07 | 33 | |
| deepseek-moe-16b-chatmax context=4k2026.05 | 25.2 | |
| Minerva-7B-instruct-v1,0max context=4k2026.05 | 16.1 | |
| Olmo3 BaseTuning Protocol=Freezing param.2026.07 | 3.67 | |
| LMK token tuningTuning Protocol=Freezing param.2026.07 | 2 |