Multi-task Language Understanding on MMLU (Loss, Accuracy, PPL)
1.33Loss (FT)MobileFineTuner
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MobileFineTunerModel=Gemma3-270M, Fine-tuning protocol=PEFT (LoRA), Sequence length=128, Batch size=8, LoRA rank=8, LoRA alpha=32, LoRA dropout=0.1, Learning rate=2e-42025.12 | 1.33 | 0.2672 | 3.81 | |
| MobileFineTunerModel=GPT2-355M, Fine-tuning protocol=PEFT (LoRA), Sequence length=128, Batch size=8, LoRA rank=8, LoRA alpha=32, LoRA dropout=0.1, Learning rate=2e-42025.12 | 1.35 | 0.2918 | 3.89 | |
| PyTorchModel=GPT2-355M, Fine-tuning protocol=PEFT (LoRA), Sequence length=128, Batch size=8, LoRA rank=8, LoRA alpha=32, LoRA dropout=0.1, Learning rate=2e-42025.12 | 1.37 | 0.3006 | 3.87 | |
| MobileFineTunerModel=GPT2-124M, Fine-tuning protocol=PEFT (LoRA), Sequence length=128, Batch size=8, LoRA rank=8, LoRA alpha=32, LoRA dropout=0.1, Learning rate=2e-42025.12 | 1.38 | 0.2353 | 3.88 | |
| PyTorchModel=GPT2-124M, Fine-tuning protocol=PEFT (LoRA), Sequence length=128, Batch size=8, LoRA rank=8, LoRA alpha=32, LoRA dropout=0.1, Learning rate=2e-42025.12 | 1.39 | 0.2545 | 4 | |
| PyTorchModel=Gemma3-270M, Fine-tuning protocol=PEFT (LoRA), Sequence length=128, Batch size=8, LoRA rank=8, LoRA alpha=32, LoRA dropout=0.1, Learning rate=2e-42025.12 | 1.45 | 0.2516 | 4.28 | |
| PyTorchModel=Gemma3-1B, Fine-tuning protocol=PEFT (LoRA), Sequence length=128, Batch size=8, LoRA rank=8, LoRA alpha=32, LoRA dropout=0.1, Learning rate=2e-42025.12 | 1.46 | 0.4147 | 3.2 | |
| MobileFineTunerModel=Gemma3-1B, Fine-tuning protocol=PEFT (LoRA), Sequence length=128, Batch size=8, LoRA rank=8, LoRA alpha=32, LoRA dropout=0.1, Learning rate=2e-42025.12 | 1.48 | 0.3902 | 3.4 | |
| PyTorchModel=Qwen2.5-0.5B, Fine-tuning protocol=PEFT (LoRA), Sequence length=128, Batch size=8, LoRA rank=8, LoRA alpha=32, LoRA dropout=0.1, Learning rate=2e-42025.12 | 1.65 | 0.4028 | 3.82 | |
| MobileFineTunerModel=Qwen2.5-0.5B, Fine-tuning protocol=PEFT (LoRA), Sequence length=128, Batch size=8, LoRA rank=8, LoRA alpha=32, LoRA dropout=0.1, Learning rate=2e-42025.12 | 1.7 | 0.3712 | 3.95 |