Multi-task Language Understanding on MMLU (Category Accuracy Breakdown)
68.71STEM AccuracyLoRA + MoLS
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| LoRA + MoLSBackbone=Qwen2.5-7B2026.05 | 68.71 | 82.87 | 66.41 | 77.21 | 73.01 | |
| APOLLOBackbone=Qwen2.5-7B2026.05 | 68.49 | 82.97 | 65.8 | 76.65 | 72.65 | |
| GaLoreBackbone=Qwen2.5-7B2026.05 | 68.32 | 82.87 | 66.08 | 76.1 | 72.55 | |
| LoRABackbone=Qwen2.5-7B2026.05 | 67.69 | 82.91 | 66.29 | 75.72 | 72.41 | |
| LoRA + MoLSBackbone=LLaMA3.2-3B2026.05 | 47.34 | 65.77 | 49.61 | 62.41 | 55.62 | |
| LoRABackbone=LLaMA3.2-3B2026.05 | 47.22 | 63.41 | 49.56 | 61.57 | 54.86 | |
| GaLoreBackbone=LLaMA3.2-3B2026.05 | 46.62 | 65.58 | 48.37 | 63.33 | 55.22 | |
| FullBackbone=LLaMA3.2-3B2026.05 | 46.55 | 65.81 | 49.16 | 63.17 | 55.48 | |
| APOLLOBackbone=LLaMA3.2-3B2026.05 | 46.16 | 65.91 | 49.03 | 62.8 | 55.29 | |
| FullBackbone=Gemma3-1B2026.05 | 27.63 | 26.84 | 25.12 | 25.14 | 26.04 | |
| LoRA + MoLSBackbone=Gemma3-1B2026.05 | 27.51 | 26.96 | 25.67 | 25.49 | 26.31 | |
| APOLLOBackbone=Gemma3-1B2026.05 | 27.47 | 27.14 | 25.61 | 25.76 | 26.38 | |
| GaLoreBackbone=Gemma3-1B2026.05 | 27.4 | 26.71 | 24.8 | 25.69 | 25.99 | |
| LoRABackbone=Gemma3-1B2026.05 | 26.61 | 25.93 | 24.78 | 25.02 | 25.48 |