Language Modeling on LAMBADA (PPL, Δ% PPL)
0.2PPL Change (%)CE + TMKL
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| CE + TMKLBackbone=Qwen2.5-7B-Instruct, Adapter=LoRA2026.05 | 0.2 | — | |
| CE + TMKLBackbone (LoRA)=Mistral-7B-v0.32026.05 | 0.2 | 24.14 | |
| SineLoRA CE + TMKLAdapter=SineLoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.05 | 0.3 | 23.5 | |
| DoRA CE + TMKLAdapter=DoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.05 | 0.3 | 23.35 | |
| CE + TMKLAdapter=RandLoRA2026.05 | 0.4 | 35.82 | |
| CE + TMKLBackbone=Qwen2.5-7B-Instruct, Adapter=SineLoRA2026.05 | 0.4 | — | |
| CE + TMKLBackbone=Qwen2.5-7B-Instruct, Adapter=DoRA2026.05 | 0.4 | — | |
| LoRA CE + TMKLAdapter=LoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.05 | 0.5 | 23.32 | |
| CE + TMKLBackbone (LoRA)=Llama-3.2-1B2026.05 | 0.5 | 38.19 | |
| CE + TMKLBackbone (LoRA)=Llama-3.1-8B2026.05 | 0.5 | 22.68 | |
| RandLoRA CE + TMKLAdapter=RandLoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.05 | 0.7 | 23.6 | |
| CE + TMKLBackbone=Qwen2.5-7B-Instruct, Adapter=RandLoRA2026.05 | 0.8 | — | |
| CE + TMKLAdapter=SineLoRA2026.05 | 1 | 36.18 | |
| CE + TMKLAdapter=LoRA2026.05 | 2 | 36.43 | |
| CE + TMKLAdapter=DoRA2026.05 | 2 | 36.44 | |
| PiSSA (CE + TMKL)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=PiSSA, Method=CE + TMKL2026.05 | 3.2 | 20.87 | |
| SineLoRA (CE)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=SineLoRA, Method=CE2026.05 | 5.5 | 22.74 | |
| SineLoRA (CE + TMKL)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=SineLoRA, Method=CE + TMKL2026.05 | 5.7 | 20.33 | |
| RandLoRA (CE + TMKL)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=RandLoRA, Method=CE + TMKL2026.05 | 6.2 | 20.23 | |
| DoRA (CE + TMKL)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=DoRA, Method=CE + TMKL2026.05 | 6.8 | 20.1 | |
| LoRA (CE + TMKL)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=LoRA, Method=CE + TMKL2026.05 | 7.3 | 19.99 | |
| DoRA (CE)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=DoRA, Method=CE2026.05 | 11 | 23.93 | |
| LoRA (CE)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=LoRA, Method=CE2026.05 | 12 | 24.22 | |
| PiSSA (CE)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=PiSSA, Method=CE2026.05 | 13 | 24.4 | |
| RandLoRA (CE)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=RandLoRA, Method=CE2026.05 | 18 | 25.43 | |
| CEAdapter=RandLoRA2026.05 | 20 | 42.87 | |
| RandLoRA CEAdapter=RandLoRA, Method=CE, Precision=BF16, Seeds=32026.05 | 25 | 29.29 | |
| no adaptationBackbone=Qwen2.5-7B-Instruct2026.05 | 25.1 | — | |
| CEBackbone=Qwen2.5-7B-Instruct, Adapter=RandLoRA2026.05 | 28.5 | — | |
| CEBackbone (LoRA)=Mistral-7B-v0.32026.05 | 31 | 31.57 | |
| SineLoRA CEAdapter=SineLoRA, Method=CE, Precision=BF16, Seeds=32026.05 | 32 | 30.93 | |
| LoRA CEAdapter=LoRA, Method=CE, Precision=BF16, Seeds=32026.05 | 33 | 31.18 | |
| DoRA CEAdapter=DoRA, Method=CE, Precision=BF16, Seeds=32026.05 | 33 | 31.16 | |
| CEBackbone (LoRA)=Llama-3.1-8B2026.05 | 34 | 30.55 | |
| CEBackbone=Qwen2.5-7B-Instruct, Adapter=SineLoRA2026.05 | 34.2 | — | |
| CEBackbone=Qwen2.5-7B-Instruct, Adapter=LoRA2026.05 | 35.8 | — | |
| CEBackbone=Qwen2.5-7B-Instruct, Adapter=DoRA2026.05 | 36.1 | — | |
| CEAdapter=SineLoRA2026.05 | 37 | 48.94 | |
| CEBackbone (LoRA)=Llama-3.2-1B2026.05 | 38 | 52.44 | |
| CEAdapter=LoRA2026.05 | 42 | 50.59 | |
| CEAdapter=DoRA2026.05 | 42 | 50.74 | |
| Base Qwen2.5-7BAdapter=None, Method=No adaptation, Precision=BF16, Seeds=32026.05 | — | 23.43 | |
| no adaptationAdapter=Base Qwen2.5-0.5B2026.05 | — | 35.68 | |
| no adaptationBackbone=Phi-3.5-mini-instruct 3.8B, Adapter=None, Method=no adaptation2026.05 | — | 21.56 | |
| no adaptationBackbone (LoRA)=Llama-3.2-1B2026.05 | — | 38 | |
| no adaptationBackbone (LoRA)=Llama-3.1-8B2026.05 | — | 22.8 | |
| no adaptationBackbone (LoRA)=Mistral-7B-v0.32026.05 | — | 24.1 |