Multi-task Language Reasoning on RLU Suite (GSM8K, ARC-C, MEDQA, BOOLQ, COLA)
90.67GSM8K AccuracyHard-Routed MoR-LoRA
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Hard-Routed MoR-LoRABackbone Model=Qwen-2.5-7B, Mixer training rank=r = 64, # Trainable Parameters=≈ 41M2026.06 | 90.67 | 89.25 | 58.13 | 88.1 | 83.41 | 81.95 | |
| LoRAMixerBackbone Model=Qwen-2.5-7B, Top-K=2, Normalization=true, # Trainable Parameters=≈ 1.211B2026.06 | 90.6 | 88.65 | 56.2 | 85.54 | 81.78 | 80.55 | |
| Hard-Routed MoR-LoRABackbone Model=Gemma-3-4B, # Trainable Parameters=≈ 72M2026.06 | 87.57 | 83.61 | 48.78 | 84.8 | 81.59 | 77.27 | |
| LoRAMixerBackbone Model=Gemma-3-4B, Top-K=2, Normalization=true, # Trainable Parameters=≈ 836M2026.06 | 86.88 | 83.1 | 47.68 | 83.43 | 81.3 | 76.48 | |
| LoRAMixerBackbone Model=Gemma-3-4B, Top-K=1, # Trainable Parameters=≈ 836M2026.06 | 86.05 | 83.7 | 46.5 | 80.94 | 80.44 | 75.53 | |
| LoRAMixerBackbone Model=Qwen-2.5-7B, Top-K=1, # Trainable Parameters=≈ 1.211B2026.06 | 74.91 | 88.99 | 57.5 | 83 | 83.13 | 77.51 |