Language Modeling on WikiText (WT)
31Relative PPL Change (%)LoRA
Evaluation results
| Method | Links | |
|---|---|---|
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+HS2026.04 | 31 | |
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+WT2026.04 | 25.2 | |
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+SST2026.04 | 23.6 | |
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+LB2026.04 | 14.5 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+Fin2026.04 | 11.1 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+WT2026.04 | 11 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+PG2026.04 | 11 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+MdQ2026.04 | 10.6 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+SST2026.04 | 10.6 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+HS2026.04 | 10.3 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+LB2026.04 | 10 | |
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+PG2026.04 | 9.8 | |
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+Fin2026.04 | 8.4 | |
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+MdQ2026.04 | 1.2 | |
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+Med2026.04 | 1 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+Med2026.04 | 0 |