Long-Context Retrieval on RULER 32K
86.65RULER 32K Retrieval ScoreHybrid-KSA
Evaluation results
| Method | Links | |
|---|---|---|
| Hybrid-KSASetting=CPT setting2026.04 | 86.65 | |
| Full AttentionSetting=CPT setting2026.04 | 84.74 | |
| KSASetting=CPT setting2026.04 | 80.3 | |
| Hybrid-SWASetting=CPT setting2026.04 | 78.94 | |
| MSA-PTTraining Strategy=from-scratch sparse pretraining2026.06 | 77.5 | |
| Hybrid-SCASetting=CPT setting2026.04 | 76.89 | |
| Hybrid-LinearSetting=CPT setting2026.04 | 76.48 | |
| MSA-CPTTraining Strategy=sparse continued pretraining2026.06 | 75.7 | |
| FullTraining Strategy=Full-Attention baseline2026.06 | 75 | |
| Hybrid-KSAsetting=Train-from-scratch2026.04 | 72.3 | |
| Hybrid-GDNsetting=Train-from-scratch2026.04 | 70.41 | |
| Fullsetting=Train-from-scratch2026.04 | 69.06 | |
| Hybrid-SWAsetting=Train-from-scratch2026.04 | 67.86 | |
| Hybrid-SCAsetting=Train-from-scratch2026.04 | 67.74 | |
| KSAsetting=Train-from-scratch2026.04 | 62.54 |