Long-context Retrieval on Needle-in-a-Haystack
100Retrieval AccuracyQwen2.5-3B-Instruct (Teacher)
Evaluation results
| Method | Links | |
|---|---|---|
| Qwen2.5-3B-Instruct (Teacher)Context length (tokens)=8,1922025.12 | 100 | |
| Hybrid student (25% softmax, 75% GDN)Context length (tokens)=8,1922025.12 | 100 | |
| Qwen2.5-3B-Instruct (Teacher)Context length (tokens)=16,3842025.12 | 100 | |
| Qwen2.5-3B-Instruct (Teacher)Context length (tokens)=32,7682025.12 | 100 | |
| Qwen2.5-3B-Instruct (Teacher)Context length (tokens)=65,5362025.12 | 100 | |
| USMoEContext Length=16.5K, Doc Depth=50%, Base Model=Qwen3-30B-A3B-Instruct2025.03 | 100 | |
| TransformerScale=50M2026.06 | 100 | |
| SISAScale=50M, ds=642026.06 | 100 | |
| TransformerScale=152M2026.06 | 100 | |
| SISAScale=152M, ds=162026.06 | 100 | |
| TransformerScale=369M, mb=42026.06 | 100 | |
| SISAScale=369M, ds=32, mb=42026.06 | 100 | |
| SISAScale=369M, ds=64, mb=4/82026.06 | 100 | |
| SISAScale=369M, ds=128, mb=42026.06 | 100 | |
| Mamba-3Scale=369M, mb=22026.06 | 100 | |
| Hybrid student (25% softmax, 75% GDN)Context length (tokens)=16,3842025.12 | 99.8 | |
| Hybrid student (25% softmax, 75% GDN)Context length (tokens)=32,7682025.12 | 99.8 | |
| SISAScale=369M, ds=32, mb=22026.06 | 99.5 | |
| Hybrid student (25% softmax, 75% GDN)Context length (tokens)=65,5362025.12 | 99.4 | |
| Mamba-3Scale=152M2026.06 | 99 | |
| Mamba-2Scale=369M, mb=42026.06 | 96.5 | |
| Qwen2.5-3B-Instruct (Teacher)Context length (tokens)=131,0722025.12 | 95.4 | |
| Mamba-3Scale=369M, mb=42026.06 | 86.5 | |
| Mamba-2Scale=152M2026.06 | 82.5 | |
| Hybrid student (25% softmax, 75% GDN)Context length (tokens)=131,0722025.12 | 68.4 | |
| Mamba-3Scale=50M2026.06 | 63.5 | |
| Original (TC)Context Length=16.5K, Doc Depth=50%, Base Model=Qwen3-30B-A3B-Instruct2025.03 | 50 | |
| ECContext Length=16.5K, Doc Depth=50%, Base Model=Qwen3-30B-A3B-Instruct2025.03 | 10 | |
| Mamba-2Scale=50M2026.06 | 0 |