Low-resource language adaptation on Bactrian-X (held-out)
1.0666Score (Indonesian)SkewGrad
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| SkewGradBase model=Llama-3.2-3B, Learning rate=2 × 10−4, Precision=bf16, Training epochs=2, Batch size=1, Gradient accumulation=64, Seeds=32026.05 | 1.0666 | 1.9037 | 1.3768 | 1.3585 | 1.3503 | 1.4112 | |
| PSOFTBase model=Llama-3.2-3B, Learning rate=2 × 10−4, Precision=bf16, Training epochs=2, Batch size=1, Gradient accumulation=64, Seeds=3, Rank=3542026.05 | 1.0557 | 1.8747 | 1.377 | 1.3511 | 1.3628 | 1.4043 | |
| PrincipalBase model=Llama-3.2-3B, Learning rate=2 × 10−4, Precision=bf16, Training epochs=2, Batch size=1, Gradient accumulation=64, Seeds=32026.05 | 1.0556 | 1.8715 | 1.3771 | 1.3513 | 1.3631 | 1.4037 | |
| GradSVDBase model=Llama-3.2-3B, Learning rate=2 × 10−4, Precision=bf16, Training epochs=2, Batch size=1, Gradient accumulation=64, Seeds=32026.05 | 1.0553 | 1.8655 | 1.3708 | 1.3463 | 1.348 | 1.3972 | |
| BPBBase model=Llama-3.2-3B, Context=Measured before fine-tuning2026.05 | 0.596 | 0.626 | 0.811 | 0.831 | 1.065 | — |