Continual Language Modeling on HOPE 16-domain stream 256M (long continual sequence)
10.2ForgettingAdaptive OGP
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Adaptive OGPRouting strategy=decoupled, Model scale=256M, Number of seeds=32026.04 | 10.2 | 36.5 | |
| Replay 0.5% (shared)Routing strategy=shared, Replay ratio=0.5%, Model scale=256M, Number of seeds=32026.04 | 12.8 | 37.8 | |
| FOPNG (shared)Routing strategy=shared, Model scale=256M, Number of seeds=32026.04 | 14.7 | 39 | |
| Adam-NSCL (shared)Routing strategy=shared, Model scale=256M, Number of seeds=32026.04 | 15 | 39.2 |