Language Modeling on FineWeb-Edu 10B (held-out last unused bin)
15.34PerplexityGDN
Evaluation results
| Method | Links | |
|---|---|---|
| GDNLayers=10, State dim=768, Mem. cells=1.97M (distributed)2026.05 | 15.34 | |
| Transformer++Layers=12, State dim=768, Mem. cells=— (KV)2026.05 | 16.05 | |
| GPN+MLayers=2, State dim=1920, Mem. cells=360K (centralized)2026.05 | 16.95 | |
| GPN+MLayers=1, State dim=2496, Mem. cells=492K (centralized)2026.05 | 18.06 | |
| GPN (no memory)Layers=1, State dim=3184, Mem. cells=02026.05 | 23.51 |