Continual Learning on 6-phase Text Domains (Prose, Python, Math, Biomedical, Chinese, JavaScript)
-0.083BWTTFGN + GPT-2 Medium (TFGN_GPT2M_FS)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| TFGN + GPT-2 Medium (TFGN_GPT2M_FS)Total params=~739 M, Regime=FS, Phases=62026.05 | -0.083 | 67 | — | |
| TFGN + GPT-2 Small (TFGN_GPT2S_FS)Total params=~398 M, Regime=FS, Phases=62026.05 | -0.109 | 67 | — | |
| TFGN + GPT-2 Medium (TFGN_GPT2M_RETROFIT)Total params=~739 M, Regime=RF, Phases=62026.05 | -0.135 | 49 | — | |
| Baseline LoRA r=256 GPT-2 Medium (RF)Total params=~355 M, Regime=RF, Phases=62026.05 | -0.393 | 70 | — | |
| Baseline Std-FT GPT-2 Medium (RF)Total params=~355 M, Regime=RF, Phases=62026.05 | -0.541 | 75 | — | |
| Baseline LoRA r=256 GPT-2 Medium (FS)Total params=~393 M, Regime=FS, Phases=62026.05 | -1.005 | 80 | — | |
| Baseline Std-FT GPT-2 Medium (FS)Total params=~355 M, Regime=FS, Phases=62026.05 | -1.17 | 85 | — |