Commonsense Reasoning on Commonsense Reasoning (test)
88.2Overall Average AccuracyFRAME
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| FRAMEBackbone=Qwen2.5-7B2026.06 | 88.2 | — | — | — | — | — | — | — | — | |
| MoABackbone=Qwen2.5-7B2026.06 | 87.5 | — | — | — | — | — | — | — | — | |
| FourierMoEBackbone=Qwen2.5-7B2026.06 | 87.4 | — | — | — | — | — | — | — | — | |
| FlyLoRABackbone=Qwen2.5-7B2026.06 | 87.3 | — | — | — | — | — | — | — | — | |
| FRAMEBackbone=LLaMA-3.1-8B2026.06 | 87.1 | — | — | — | — | — | — | — | — | |
| MoABackbone=LLaMA-3.1-8B2026.06 | 86.3 | — | — | — | — | — | — | — | — | |
| FourierMoEBackbone=LLaMA-3.1-8B2026.06 | 86.2 | — | — | — | — | — | — | — | — | |
| FlyLoRABackbone=LLaMA-3.1-8B2026.06 | 86.1 | — | — | — | — | — | — | — | — | |
| LoRABackbone=Qwen2.5-7B2026.06 | 85.9 | — | — | — | — | — | — | — | — | |
| LoRABackbone=LLaMA-3.1-8B2026.06 | 84.5 | — | — | — | — | — | — | — | — | |
| FSLoRABackbone=LLaMA-3.2-3B-Instruct, GPU hours=44.3h2025.01 | 79.4 | 69.3 | 84 | 76.8 | 82.2 | 79.1 | 87.2 | 76.1 | 80.7 | |
| FSLoRANumber of clients=100, Distribution=Heavy-tailed, Model=LLaMA-3.2-3B-Instruct2025.01 | 78.8 | 70.13 | 83.57 | 76.51 | 79.9 | 78.93 | 87.54 | 74.4 | 79.4 | |
| FlexLoRABackbone=LLaMA-3.2-3B-Instruct, GPU hours=68.3h2025.01 | 77.4 | 68.6 | 81 | 75.9 | 79.4 | 77.9 | 86.7 | 74.2 | 75.8 | |
| FlexLoRANumber of clients=100, Distribution=Heavy-tailed, Model=LLaMA-3.2-3B-Instruct2025.01 | 76.53 | 69.03 | 80.47 | 74.16 | 75.27 | 73.8 | 87.54 | 73.38 | 78.6 | |
| RAVANBackbone=LLaMA-3.2-3B-Instruct, GPU hours=51.2h2025.01 | 74.7 | 65.7 | 77.1 | 74.8 | 76.7 | 73.8 | 81.6 | 71.9 | 75.6 | |
| HeteroLoRABackbone=LLaMA-3.2-3B-Instruct, GPU hours=43.7h2025.01 | 74.6 | 65.8 | 80.9 | 73.8 | 73 | 72 | 86.6 | 73.4 | 71.4 | |
| FLoRABackbone=LLaMA-3.2-3B-Instruct, GPU hours=49.8h2025.01 | 74.6 | 65.8 | 80.5 | 76.1 | 77.2 | 71.5 | 83.1 | 68.3 | 74.2 | |
| HeteroLoRANumber of clients=100, Distribution=Heavy-tailed, Model=LLaMA-3.2-3B-Instruct2025.01 | 73.55 | 62.57 | 79.38 | 74.1 | 68.07 | 69.69 | 86.24 | 71.76 | 76.6 | |
| FLoRANumber of clients=100, Distribution=Heavy-tailed, Model=LLaMA-3.2-3B-Instruct2025.01 | 73.44 | 67.1 | 78.94 | 72.21 | 71.67 | 70.8 | 83.25 | 69.97 | 73.6 | |
| PiCa#Params=5.37M, Backbone=Gemma-2B, r=2562025.05 | 67.6 | 63.91 | 75.57 | 64.38 | 71.75 | 60.62 | 77.44 | 58.7 | 68.4 | |
| Full-FT#Params=2.5B, Backbone=Gemma-2B2025.05 | 67.45 | 63.57 | 74.1 | 65.86 | 70 | 61.95 | 75.36 | 59.72 | 69 | |
| SVFT_B#Params=6.35M, Backbone=Gemma-2B, d=162025.05 | 65.86 | 63.42 | 73.72 | 63.86 | 71.21 | 59.58 | 73.69 | 54.77 | 66.6 | |
| DoRA#Params=13.5M, Backbone=Gemma-2B, r=162025.05 | 64.4 | 62.87 | 73.93 | 65.34 | 53.16 | 55.51 | 76.43 | 59.55 | 68.4 | |
| LoRA#Params=26.2M, Backbone=Gemma-2B, r=322025.05 | 62.43 | 63.11 | 73.44 | 63.2 | 47.79 | 52.95 | 74.78 | 57.16 | 67 | |
| PiCa#Params=0.67M, Backbone=Gemma-2B, r=322025.05 | 58.82 | 62.11 | 71.76 | 60.13 | 36.49 | 50.59 | 73.74 | 52.56 | 63.2 | |
| LoRA#Params=0.82M, Backbone=Gemma-2B, r=12025.05 | 55.81 | 62.2 | 69.31 | 56.24 | 32.47 | 51.53 | 69.52 | 48.8 | 56.4 | |
| SVFT_P#Params=0.19M, Backbone=Gemma-2B2025.05 | 55.81 | 62.26 | 70.18 | 56.7 | 32.47 | 47.04 | 69.31 | 50.08 | 58.4 | |
| DoRA#Params=1.19M, Backbone=Gemma-2B, r=12025.05 | 55.52 | 62.17 | 68.77 | 55.93 | 32.95 | 51.22 | 68.81 | 48.72 | 55.6 | |
| VeRA#Params=0.66M, Backbone=Gemma-2B, r=20482025.05 | 50.23 | 62.11 | 64.31 | 49.18 | 32 | 50.74 | 58.08 | 42.83 | 42.6 | |
| BOFT#Params=1.22M, Backbone=Gemma-2B, b=8, m=22025.05 | 49.22 | 59.23 | 63.65 | 47.9 | 29.93 | 50.35 | 59.04 | 42.66 | 41 | |
| Base modelBackbone=LLaMA-3.2-3B-Instruct2025.01 | 43.6 | 47.2 | 21.3 | 54.4 | 55.8 | 4.6 | 61.4 | 51.5 | 52.4 |