Language Modeling on text (val)
2.253Text NLL (Validation)SDM
Evaluation Results
| Method | Links | |
|---|---|---|
| SDMScale=8B (L13), Local layer=SWA, Global layer=SDM2026.07 | 2.253 | |
| FullAttnScale=8B (L13), Local layer=SWA, Global layer=FullAttn2026.07 | 2.285 | |
| GDNScale=8B (L13), Local layer=SWA, Global layer=GDN2026.07 | 2.298 | |
| SDMScale=1.4B (L08), Local layer=SWA, Global layer=SDM2026.07 | 2.654 | |
| FullAttnScale=1.4B (L08), Local layer=SWA, Global layer=FullAttn2026.07 | 2.66 | |
| GDNScale=1.4B (L08), Local layer=SWA, Global layer=GDN2026.07 | 2.683 | |
| Mamba2Scale=1.4B (L08), Local layer=SWA, Global layer=Mamba22026.07 | 2.686 |