Language Modeling on LAMBADA (Retention Drift & Drift Prevention)
2Retention Drift (%)CE + TMKL
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| CE + TMKLBase Model=Qwen2.5-0.5B, Adapter Type=LoRA, Stability Hyperparameter (λ)=12026.05 | 2 | 95 | |
| LwF / Full-KLBase Model=Qwen2.5-0.5B, Adapter Type=LoRA, Stability Hyperparameter (λ)=12026.05 | 12 | 71 | |
| STABLEBase Model=Qwen2.5-0.5B, Adapter Type=LoRA2026.05 | 14 | 67 | |
| O-LoRABase Model=Qwen2.5-0.5B, Adapter Type=LoRA2026.05 | 25 | 40 | |
| EWCBase Model=Qwen2.5-0.5B, Adapter Type=LoRA2026.05 | 31 | 26 | |
| L2-SPBase Model=Qwen2.5-0.5B, Adapter Type=LoRA2026.05 | 35 | 17 | |
| LoRA + CE (no regularizer)Base Model=Qwen2.5-0.5B, Adapter Type=LoRA2026.05 | 42 | — |