Language Modeling on LAMBADA (Delta %)
43.3Delta (%)Progressive Linearization
Evaluation Results
| Method | Links | |
|---|---|---|
| Progressive LinearizationStep=8002026.03 | 43.3 | |
| Progressive LinearizationStep=10002026.03 | 40.8 | |
| Progressive LinearizationStep=14002026.03 | 39.8 | |
| Progressive LinearizationStep=4002026.03 | 39.2 | |
| Progressive LinearizationStep=20002026.03 | 38.6 | |
| Progressive LinearizationStep=0 (post-lin)2026.03 | 36.7 | |
| Progressive LinearizationStep=2002026.03 | 23.7 | |
| LoRALR=10−2, Params=92K2026.04 | 7.1 | |
| LoRALR=10−3, Params=92K2026.04 | 3 | |
| GAIN-FFNLR=3×10−3, Params=230K2026.04 | 2.9 | |
| LoRAModel=Mistral-7B, Params=262K, Learning Rate=10−32026.04 | 2.7 | |
| GAIN-FFNLR=10−3, Params=230K2026.04 | 2 | |
| GAINLR=3×10−3, Params=46K2026.04 | 0.9 | |
| LoRALR=3×10−4, Params=92K2026.04 | 0.7 | |
| LoRAModel=Llama-2-70B, Params=1.3M, Learning Rate=10−32026.04 | 0.5 | |
| LoRAModel=Llama-2-70B, Params=1.3M, Learning Rate=3×10−42026.04 | 0.5 | |
| GAINLR=10−3, Params=46K2026.04 | 0.4 | |
| GAINModel=Mistral-7B, Params=131K2026.04 | 0.4 | |
| GAINModel=Llama-2-13B, Params=205K2026.04 | 0.3 | |
| LoRAModel=Llama-2-13B, Params=410K, Learning Rate=10−32026.04 | 0.2 | |
| LoRAModel=Llama-2-13B, Params=410K, Learning Rate=3×10−42026.04 | 0.2 | |
| GAINLR=3×10−4, Params=46K2026.04 | 0.1 | |
| LoRALR=10−4, Params=92K2026.04 | 0.1 | |
| GAINModel=Llama-2-70B, Params=655K2026.04 | 0.1 | |
| LoRAModel=Mistral-7B, Params=262K, Learning Rate=3×10−42026.04 | 0 |