Language Modeling on WT-103
8.28PerplexityCE + TMKL
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| CE + TMKLBackbone (LoRA)=Llama-3.1-8B2026.05 | 8.28 | 2 | |
| no adaptationBackbone (LoRA)=Llama-3.1-8B2026.05 | 8.45 | — | |
| CE + TMKLBackbone (LoRA)=Mistral-7B-v0.32026.05 | 8.81 | 1 | |
| no adaptationBackbone (LoRA)=Mistral-7B-v0.32026.05 | 8.9 | — | |
| CEBackbone (LoRA)=Llama-3.1-8B2026.05 | 10.05 | 19 | |
| PiSSA (CE + TMKL)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=PiSSA, Method=CE + TMKL2026.05 | 10.19 | 2.8 | |
| LoRA (CE + TMKL)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=LoRA, Method=CE + TMKL2026.05 | 10.25 | 2.2 | |
| DoRA (CE + TMKL)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=DoRA, Method=CE + TMKL2026.05 | 10.29 | 1.8 | |
| CEBackbone (LoRA)=Mistral-7B-v0.32026.05 | 10.41 | 17 | |
| no adaptationBackbone=Phi-3.5-mini-instruct 3.8B, Adapter=None, Method=no adaptation2026.05 | 10.48 | — | |
| RandLoRA (CE + TMKL)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=RandLoRA, Method=CE + TMKL2026.05 | 10.5 | 0.2 | |
| SineLoRA (CE + TMKL)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=SineLoRA, Method=CE + TMKL2026.05 | 10.69 | 2 | |
| SineLoRA (CE)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=SineLoRA, Method=CE2026.05 | 11.28 | 7.7 | |
| LoRA (CE)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=LoRA, Method=CE2026.05 | 12.17 | 16 | |
| DoRA (CE)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=DoRA, Method=CE2026.05 | 12.17 | 16 | |
| CE + TMKLBackbone (LoRA)=Llama-3.2-1B2026.05 | 12.31 | 1.5 | |
| PiSSA (CE)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=PiSSA, Method=CE2026.05 | 12.42 | 19 | |
| no adaptationBackbone (LoRA)=Llama-3.2-1B2026.05 | 12.5 | — | |
| RandLoRA (CE)Backbone=Phi-3.5-mini-instruct 3.8B, Adapter=RandLoRA, Method=CE2026.05 | 13.22 | 26 | |
| CEBackbone (LoRA)=Llama-3.2-1B2026.05 | 15.5 | 24 |