Commonsense Reasoning on Commonsense 8 Sub-Tasks
88.22Accuracy (8 Sub-Tasks)LR-LoRA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| LR-LoRABackbone=LLaMA3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 88.22 | — | |
| LR-LoRABackbone=Phi-3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 87.3 | — | |
| ElaLoRABackbone=LLaMA3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 87.05 | — | |
| ElaLoRABackbone=Phi-3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 86.2 | — | |
| LR-LoRABackbone=LLaMA3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 85.99 | — | |
| AdaLoRABackbone=LLaMA3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 85.88 | — | |
| RandLoRABackbone=LLaMA3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 85.59 | — | |
| SineLoRABackbone=LLaMA3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 85.42 | — | |
| DyLoRABackbone=LLaMA3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 85.4 | — | |
| AdaLoRABackbone=Phi-3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 85.35 | — | |
| LoRABackbone=LLaMA3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 85.24 | — | |
| RandLoRABackbone=Phi-3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 85.22 | — | |
| DyLoRABackbone=Phi-3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 85.05 | — | |
| SineLoRABackbone=Phi-3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 85.01 | — | |
| LoRABackbone=Phi-3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 84.96 | — | |
| ElaLoRABackbone=LLaMA3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 84.55 | — | |
| LR-LoRABackbone=Phi-3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 83.9 | — | |
| AdaLoRABackbone=LLaMA3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 83.5 | — | |
| DyLoRABackbone=LLaMA3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 83.25 | — | |
| LoRABackbone=LLaMA3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 83.09 | — | |
| ElaLoRABackbone=Phi-3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 82.55 | — | |
| NoLABackbone=Phi-3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 82.33 | — | |
| RandLoRABackbone=Phi-3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 82.33 | — | |
| SineLoRABackbone=LLaMA3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 81.9 | — | |
| VeRABackbone=LLaMA3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 81.65 | — | |
| AdaLoRABackbone=Phi-3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 81.45 | — | |
| VeRABackbone=Phi-3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 81.4 | — | |
| RandLoRABackbone=LLaMA3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 81.31 | — | |
| NoLABackbone=LLaMA3-8B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 81.16 | — | |
| SineLoRABackbone=Phi-3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 81.12 | — | |
| DyLoRABackbone=Phi-3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 80.85 | — | |
| NoLABackbone=Phi-3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 80.35 | — | |
| LoRABackbone=Phi-3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 80.34 | — | |
| VeRABackbone=Phi-3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 78.63 | — | |
| VeRABackbone=LLaMA3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 77.14 | — | |
| NoLABackbone=LLaMA3-8B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 76.87 | — | |
| LoRARatio=0%, # Params=0.45B2026.03 | 61.4 | 55.5 | |
| LR-LoRABackbone=Qwen2-0.5B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 61.05 | — | |
| OriginalRatio=0%, # Params=-2026.03 | 60.6 | 37.7 | |
| ElaLoRABackbone=Qwen2-0.5B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 59.15 | — | |
| MoBERatio=30%, # Params=0.45B2026.03 | 58.3 | 50.4 | |
| AdaLoRABackbone=Qwen2-0.5B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 58.2 | — | |
| RandLoRABackbone=Qwen2-0.5B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 57.86 | — | |
| DyLoRABackbone=Qwen2-0.5B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 57.6 | — | |
| SineLoRABackbone=Qwen2-0.5B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 57.55 | — | |
| LoRABackbone=Qwen2-0.5B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 57.34 | — | |
| Replace (w. shared)Ratio=30%, # Params=0.45B2026.03 | 56.7 | 52.3 | |
| LightMoERatio=30%, # Params=0.45B2026.03 | 56.3 | 55.3 | |
| LR-LoRABackbone=Qwen2-0.5B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 55.88 | — | |
| Replace (w/o shared)Ratio=30%, # Params=0.45B2026.03 | 55.6 | 52.6 | |
| HC-SMoERatio=30%, # Params=0.45B2026.03 | 55.5 | 46.8 | |
| MC-SMoERatio=30%, # Params=0.45B2026.03 | 55.1 | 53.4 | |
| Interdomain AttentionZero-shot=true, Model Size=1.3B, Evaluation Harness=lm-evaluation-harness2026.05 | 54.54 | — | |
| Full FTRatio=0%, # Params=6.92B2026.03 | 54.2 | 56.3 | |
| ElaLoRABackbone=Qwen2-0.5B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 54.02 | — | |
| MoBERatio=40%, # Params=0.45B2026.03 | 54 | 43 | |
| Replace (w. shared)Ratio=40%, # Params=0.45B2026.03 | 53.9 | 49.3 | |
| LightMoERatio=40%, # Params=0.45B2026.03 | 53.2 | 53 | |
| AdaLoRABackbone=Qwen2-0.5B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 53.15 | — | |
| RandLoRABackbone=Qwen2-0.5B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 52.89 | — | |
| DyLoRABackbone=Qwen2-0.5B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 52.75 | — | |
| SineLoRABackbone=Qwen2-0.5B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 52.51 | — | |
| Replace (w/o shared)Ratio=40%, # Params=0.45B2026.03 | 52.5 | 50.1 | |
| LoRABackbone=Qwen2-0.5B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 52.34 | — | |
| VeRABackbone=Qwen2-0.5B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 51.78 | — | |
| SoftmaxZero-shot=true, Model Size=1.3B, Evaluation Harness=lm-evaluation-harness2026.05 | 51.51 | — | |
| HC-SMoERatio=40%, # Params=0.45B2026.03 | 51.5 | 38.4 | |
| MC-SMoERatio=40%, # Params=0.45B2026.03 | 50.4 | 47.3 | |
| S4D-onlyZero-shot=true, Model Size=1.3B, Evaluation Harness=lm-evaluation-harness2026.05 | 49.44 | — | |
| Replace (w. shared)Ratio=50%, # Params=0.45B2026.03 | 49.3 | 41.6 | |
| VeRABackbone=Qwen2-0.5B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 48.12 | — | |
| LightMoERatio=50%, # Params=0.45B2026.03 | 48.1 | 48.1 | |
| HC-SMoERatio=50%, # Params=0.45B2026.03 | 47.8 | 34 | |
| NoLABackbone=Qwen2-0.5B, Data Regime (Samples)=170k, Rank (r)=322026.06 | 47.4 | — | |
| Replace (w/o shared)Ratio=50%, # Params=0.45B2026.03 | 47.1 | 44.3 | |
| MoBERatio=50%, # Params=0.45B2026.03 | 46.4 | 29.9 | |
| MC-SMoERatio=50%, # Params=0.45B2026.03 | 44.8 | 42.5 | |
| NoLABackbone=Qwen2-0.5B, Data Regime (Samples)=15k, Rank (r)=322026.06 | 42.55 | — | |
| MC-SMoE*Ratio=40%, # Params=1.65B2026.03 | 40.3 | 50.2 | |
| MC-SMoE*Ratio=50%, # Params=1.65B2026.03 | 40 | 45.3 |