List your company
ResearchFormerly WizWand

Language Modeling on LAMBADA (PPL, Δ% PPL)

0.2PPL Change (%)CE + TMKL
-1.4729.81421.132.386May 28, 2026
Updated 5mo ago

Evaluation results

MethodLinks
CE + TMKLBackbone=Qwen2.5-7B-Instruct, Adapter=LoRA2026.050.2—
CE + TMKLBackbone (LoRA)=Mistral-7B-v0.32026.050.224.14
SineLoRA CE + TMKLAdapter=SineLoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.050.323.5
DoRA CE + TMKLAdapter=DoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.050.323.35
CE + TMKLAdapter=RandLoRA2026.050.435.82
CE + TMKLBackbone=Qwen2.5-7B-Instruct, Adapter=SineLoRA2026.050.4—
CE + TMKLBackbone=Qwen2.5-7B-Instruct, Adapter=DoRA2026.050.4—
LoRA CE + TMKLAdapter=LoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.050.523.32
CE + TMKLBackbone (LoRA)=Llama-3.2-1B2026.050.538.19
CE + TMKLBackbone (LoRA)=Llama-3.1-8B2026.050.522.68
RandLoRA CE + TMKLAdapter=RandLoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.050.723.6
CE + TMKLBackbone=Qwen2.5-7B-Instruct, Adapter=RandLoRA2026.050.8—
CE + TMKLAdapter=SineLoRA2026.05136.18
CE + TMKLAdapter=LoRA2026.05236.43
CE + TMKLAdapter=DoRA2026.05236.44
PiSSA (CE + TMKL)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=PiSSA, Method=CE + TMKL2026.053.220.87
SineLoRA (CE)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=SineLoRA, Method=CE2026.055.522.74
SineLoRA (CE + TMKL)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=SineLoRA, Method=CE + TMKL2026.055.720.33
RandLoRA (CE + TMKL)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=RandLoRA, Method=CE + TMKL2026.056.220.23
DoRA (CE + TMKL)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=DoRA, Method=CE + TMKL2026.056.820.1
LoRA (CE + TMKL)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=LoRA, Method=CE + TMKL2026.057.319.99
DoRA (CE)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=DoRA, Method=CE2026.051123.93
LoRA (CE)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=LoRA, Method=CE2026.051224.22
PiSSA (CE)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=PiSSA, Method=CE2026.051324.4
RandLoRA (CE)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=RandLoRA, Method=CE2026.051825.43
CEAdapter=RandLoRA2026.052042.87
RandLoRA CEAdapter=RandLoRA, Method=CE, Precision=BF16, Seeds=32026.052529.29
no adaptationBackbone=Qwen2.5-7B-Instruct2026.0525.1—
CEBackbone=Qwen2.5-7B-Instruct, Adapter=RandLoRA2026.0528.5—
CEBackbone (LoRA)=Mistral-7B-v0.32026.053131.57
SineLoRA CEAdapter=SineLoRA, Method=CE, Precision=BF16, Seeds=32026.053230.93
LoRA CEAdapter=LoRA, Method=CE, Precision=BF16, Seeds=32026.053331.18
DoRA CEAdapter=DoRA, Method=CE, Precision=BF16, Seeds=32026.053331.16
CEBackbone (LoRA)=Llama-3.1-8B2026.053430.55
CEBackbone=Qwen2.5-7B-Instruct, Adapter=SineLoRA2026.0534.2—
CEBackbone=Qwen2.5-7B-Instruct, Adapter=LoRA2026.0535.8—
CEBackbone=Qwen2.5-7B-Instruct, Adapter=DoRA2026.0536.1—
CEAdapter=SineLoRA2026.053748.94
CEBackbone (LoRA)=Llama-3.2-1B2026.053852.44
CEAdapter=LoRA2026.054250.59
CEAdapter=DoRA2026.054250.74
Base Qwen2.5-7BAdapter=None, Method=No adaptation, Precision=BF16, Seeds=32026.05—23.43
no adaptationAdapter=Base Qwen2.5-0.5B2026.05—35.68
no adaptationBackbone=Phi-3.5-mini-instruct 3.8B, Adapter=None, Method=no adaptation2026.05—21.56
no adaptationBackbone (LoRA)=Llama-3.2-1B2026.05—38
no adaptationBackbone (LoRA)=Llama-3.1-8B2026.05—22.8
no adaptationBackbone (LoRA)=Mistral-7B-v0.32026.05—24.1