List your company
ResearchFormerly WizWand

Language Modeling on LAMBADA (Delta %)

43.3Delta (%)Progressive Linearization
-1.7329.95921.6533.341Mar 3, 2026Mar 8, 2026Mar 14, 2026Mar 20, 2026Mar 25, 2026Mar 31, 2026Apr 6, 2026
Updated 6mo ago

Evaluation results

MethodLinks
Progressive LinearizationStep=8002026.0343.3
Progressive LinearizationStep=10002026.0340.8
Progressive LinearizationStep=14002026.0339.8
Progressive LinearizationStep=4002026.0339.2
Progressive LinearizationStep=20002026.0338.6
Progressive LinearizationStep=0 (post-lin)2026.0336.7
Progressive LinearizationStep=2002026.0323.7
LoRALR=10−2, Params=92K2026.047.1
LoRALR=10−3, Params=92K2026.043
GAIN-FFNLR=3×10−3, Params=230K2026.042.9
LoRAModel=Mistral-7B, Params=262K, Learning Rate=10−32026.042.7
GAIN-FFNLR=10−3, Params=230K2026.042
GAINLR=3×10−3, Params=46K2026.040.9
LoRALR=3×10−4, Params=92K2026.040.7
LoRAModel=Llama-2-70B, Params=1.3M, Learning Rate=10−32026.040.5
LoRAModel=Llama-2-70B, Params=1.3M, Learning Rate=3×10−42026.040.5
GAINLR=10−3, Params=46K2026.040.4
GAINModel=Mistral-7B, Params=131K2026.040.4
GAINModel=Llama-2-13B, Params=205K2026.040.3
LoRAModel=Llama-2-13B, Params=410K, Learning Rate=10−32026.040.2
LoRAModel=Llama-2-13B, Params=410K, Learning Rate=3×10−42026.040.2
GAINLR=3×10−4, Params=46K2026.040.1
LoRALR=10−4, Params=92K2026.040.1
GAINModel=Llama-2-70B, Params=655K2026.040.1
LoRAModel=Mistral-7B, Params=262K, Learning Rate=3×10−42026.040