Language Modeling on LAMBADA Epoch 2 (Accuracy, Perplexity)
27.9LAM AccuracyNC-FFN + decay (learned γ, λ)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| NC-FFN + decay (learned γ, λ)Layers=12L, Parameter matching=true, Seed=Single2026.06 | 27.9 | 87.6 | |
| GELU baselineLayers=12L, Parameter matching=true, Seed=Single2026.06 | 27.5 | 78.1 | |
| NC-FFN ({∩, \})Layers=12L, Parameter matching=true, Seed=Single2026.06 | 27.3 | 85.5 | |
| NC-FFN + quant (no forgetting)Layers=12L, Parameter matching=true, Seed=Single2026.06 | 27.1 | 94 | |
| NC-FFN + decay + gateLayers=12L, Parameter matching=true, Seed=Single2026.06 | 26.4 | 96.3 |