Long-context Retrieval on NIAH Single-1 (Accuracies at 1024, 2048, 4096)
100Accuracy (1024)Transformer
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| TransformerModel (1.5B)=1.5B2026.05 | 100 | 100 | 0 | |
| GDNModel (1.5B)=1.5B2026.05 | 100 | 100 | 99.8 | |
| Mamba-2Model (1.5B)=1.5B2026.05 | 100 | 99.6 | 62 | |
| Looped TransformerModel (1.5B)=1.5B, Looped=true2026.05 | 100 | 100 | 0 | |
| Looped GDNModel (1.5B)=1.5B, Looped=true2026.05 | 100 | 100 | 99.8 | |
| Looped Mamba-2Model (1.5B)=1.5B, Looped=true2026.05 | 100 | 100 | 65.7 | |
| Looped Hybrid (GDN+DSA)Model (1.5B)=1.5B, Looped=true2026.05 | 100 | 100 | 91.4 | |
| Looped Hybrid (Full+GDN)Model (1.5B)=1.5B, Looped=true2026.05 | 100 | 100 | 93.5 |