Language Modeling on WikiText-103 (PPL and PPL Delta)
8.53PerplexityLoRA CE + TMKL
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| LoRA CE + TMKLAdapter=LoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.05 | 8.53 | 2 | |
| DoRA CE + TMKLAdapter=DoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.05 | 8.55 | 1.5 | |
| SineLoRA CE + TMKLAdapter=SineLoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.05 | 8.6 | 1 | |
| Base Qwen2.5-7BAdapter=None, Method=No adaptation, Precision=BF16, Seeds=32026.05 | 8.69 | — | |
| RandLoRA CE + TMKLAdapter=RandLoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.05 | 8.7 | 0.1 | |
| RandLoRA CEAdapter=RandLoRA, Method=CE, Precision=BF16, Seeds=32026.05 | 9.99 | 15 | |
| DoRA CEAdapter=DoRA, Method=CE, Precision=BF16, Seeds=32026.05 | 10.25 | 18 | |
| LoRA CEAdapter=LoRA, Method=CE, Precision=BF16, Seeds=32026.05 | 10.28 | 18 | |
| SineLoRA CEAdapter=SineLoRA, Method=CE, Precision=BF16, Seeds=32026.05 | 10.43 | 20 |