Continual Language Modeling on HOPE 8-domain, clean-regime 256M (3 seeds)
35.8Avg PPLAdaptive-OGP
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Adaptive-OGPType=OptSt2026.04 | 35.8 | 9.1 | 44.9 | 27.8 | |
| OGPType=OptSt, Model scale=256M HOPE, Seeds=3, SVD rank=Matched, Optimizer=Matched, Update frequency=Matched2026.04 | 35.8 | 9.1 | 44.9 | 27.8 | |
| Replay 0.5%Type=ER2026.04 | 36.2 | 9.8 | 45 | 22.2 | |
| Replay 0.5%Type=ER, Model scale=256M HOPE, Seeds=32026.04 | 36.2 | 9.8 | 45 | 22.2 | |
| FOPNGType=Proj2026.04 | 36.8 | 10.3 | 45.8 | 18.3 | |
| Adam-NSCLType=Proj2026.04 | 36.8 | 10.7 | 45.3 | 15.1 | |
| FOPNGType=Proj, Model scale=256M HOPE, Seeds=3, SVD rank=Matched, Optimizer=Matched, Update frequency=Matched2026.04 | 36.8 | 10.3 | 45.8 | 18.3 | |
| Adam-NSCLType=Proj, Model scale=256M HOPE, Seeds=3, SVD rank=Matched, Optimizer=Matched, Update frequency=Matched2026.04 | 36.8 | 10.7 | 45.3 | 15.1 | |
| A-GEMType=Mem, Model scale=256M HOPE, Seeds=32026.04 | 36.9 | 10.6 | 45.5 | 15.9 | |
| TRGPType=Proj, Model scale=256M HOPE, Seeds=3, SVD rank=Matched, Optimizer=Matched, Update frequency=Matched2026.04 | 36.9 | 10.4 | 45.3 | 17.5 | |
| ROGOType=Proj, Model scale=256M HOPE, Seeds=3, SVD rank=Matched, Optimizer=Matched, Update frequency=Matched2026.04 | 37.1 | 10.5 | 45.6 | 16.7 | |
| SGPType=Proj2026.04 | 37.4 | 10.8 | 45.4 | 14.3 | |
| MoFOType=OptSt2026.04 | 37.4 | 10.9 | 45.1 | 13.5 | |
| SGPType=Proj, Model scale=256M HOPE, Seeds=3, SVD rank=Matched, Optimizer=Matched, Update frequency=Matched2026.04 | 37.4 | 10.8 | 45.4 | 14.3 | |
| MoFOType=OptSt, Model scale=256M HOPE, Seeds=32026.04 | 37.4 | 10.9 | 45.1 | 13.5 | |
| EWCType=Reg2026.04 | 37.9 | 11.5 | 46.1 | 8.7 | |
| EWCType=Reg, Model scale=256M HOPE, Seeds=32026.04 | 37.9 | 11.5 | 46.1 | 8.7 | |
| Vanilla HOPEType=—2026.04 | 38.7 | 12.6 | 44.8 | — | |
| Vanilla HOPEModel scale=256M HOPE, Seeds=32026.04 | 38.7 | 12.6 | 44.8 | — |