Commonsense Reasoning on BoolQ, PIQA, SIQA, HellaS., WinoG., ARC-e, ARC-c, OBQA
89.69BoolQ AccuracyLoRA-GA
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| LoRA-GABackbone=LLaMA3-8B, Rank (r)=82025.10 | 89.69 | 84.98 | 61 | 86.58 | 85.32 | 86.11 | 62.29 | 61.8 | 77.22 | — | |
| LoRABackbone=LLaMA3-8B, Rank (r)=322025.10 | 89.69 | 85.47 | 61.72 | 86.76 | 83.35 | 87.08 | 64.08 | 62.2 | 77.54 | — | |
| ReLoRABackbone=LLaMA3-8B, Rank (r)=82025.10 | 89.2 | 85.64 | 60.13 | 85.99 | 85.24 | 86.95 | 63.14 | 61.8 | 77.26 | — | |
| ScaLoRABackbone=LLaMA3-8B, Rank (r)=82025.10 | 89.2 | 86.18 | 61.82 | 86.51 | 84.53 | 86.57 | 65.61 | 62.4 | 77.85 | — | |
| ScaLoRA-IBackbone=LLaMA3-8B, Rank (r)=82025.10 | 89.14 | 86.07 | 62.33 | 86.48 | 83.35 | 86.53 | 64.68 | 62 | 77.57 | — | |
| LoRABackbone=LLaMA3-8B, Rank (r)=82025.10 | 88.99 | 85.09 | 60.95 | 86.09 | 82.64 | 86.62 | 62.29 | 62 | 76.83 | — | |
| HiRABackbone=LLaMA3-8B, Rank (r)=82025.10 | 88.87 | 86.07 | 60.64 | 86.11 | 84.53 | 87.12 | 63.91 | 62.4 | 77.46 | — | |
| MoRABackbone=LLaMA3-8B, Rank (r)=82025.10 | 88.56 | 86.18 | 60.29 | 86.69 | 82.4 | 87.79 | 64.08 | 62.2 | 77.27 | — | |
| LoRABackbone=LLaMA2-7B, Rank (r)=322025.10 | 88.29 | 82.7 | 60.54 | 83.15 | 82 | 82.79 | 57.68 | 59 | 74.52 | — | |
| LoRA-GABackbone=LLaMA2-7B, Rank (r)=82025.10 | 87.92 | 83.03 | 60.13 | 83.3 | 82.87 | 83.25 | 56.83 | 58.4 | 74.34 | — | |
| ReLoRABackbone=LLaMA2-7B, Rank (r)=82025.10 | 87.8 | 82.48 | 60.08 | 83.23 | 82.56 | 82.95 | 58.11 | 58 | 74.4 | — | |
| ScaLoRABackbone=LLaMA2-7B, Rank (r)=82025.10 | 87.77 | 82.43 | 60.08 | 83.43 | 82.08 | 83.54 | 58.11 | 58.6 | 74.51 | — | |
| HiRABackbone=LLaMA2-7B, Rank (r)=82025.10 | 87.71 | 82.97 | 59.83 | 83.38 | 81.69 | 82.83 | 55.55 | 57.6 | 73.95 | — | |
| ScaLoRA-IBackbone=LLaMA2-7B, Rank (r)=82025.10 | 87.58 | 82.26 | 60.49 | 83.52 | 81.69 | 83.75 | 58.53 | 60.2 | 74.75 | — | |
| MoRABackbone=LLaMA2-7B, Rank (r)=82025.10 | 87.49 | 82.54 | 59.88 | 82.56 | 79.08 | 83.59 | 58.02 | 57.4 | 73.82 | — | |
| LoRABackbone=LLaMA2-7B, Rank (r)=82025.10 | 87.4 | 81.66 | 59.16 | 82.45 | 79.48 | 82.91 | 57.59 | 58.4 | 73.63 | — | |
| PARABackbone=Gemma3-4B2026.04 | 84.4 | 74.8 | 50.2 | 48.1 | 74.2 | 58.2 | 47.6 | 45.4 | 60.4 | — | |
| LoRABackbone=Gemma3-4B2026.04 | 83.6 | 73.2 | 49.8 | 47.8 | 74 | 54.4 | 43.6 | 43.2 | 58.7 | — | |
| GoRABackbone=Gemma3-4B2026.04 | 83 | 67.6 | 51.8 | 46.8 | 70.4 | 46.7 | 43.2 | 40.4 | 56.2 | — | |
| FFA-LoRA2026.03 | 82.88 | 78.67 | 51.85 | 76.3 | 74.27 | 81 | 49.79 | 45.2 | 67.49 | — | |
| FedMomentum2026.03 | 82.77 | 79.03 | 54.48 | 77.54 | 75.81 | 82.16 | 51.37 | 49 | 69.02 | — | |
| FedIT2026.03 | 82.1 | 78.35 | 53.23 | 76.63 | 74.3 | 80.52 | 50.64 | 47.7 | 67.93 | — | |
| RoLoRA2026.03 | 81.89 | 77.15 | 48.56 | 76.12 | 69.61 | 78.87 | 50.68 | 42.2 | 65.64 | — | |
| DoRABackbone=Gemma3-4B2026.04 | 81.8 | 68.2 | 52.6 | 47.3 | 70.2 | 58.9 | 42.8 | 39.2 | 57.6 | — | |
| AdaLoRABackbone=Gemma3-4B2026.04 | 81.2 | 70.3 | 48.4 | 46.5 | 71.6 | 52.8 | 43.4 | 42.4 | 57.1 | — | |
| SoRABackbone=Gemma3-4B2026.04 | 80.4 | 69.1 | 50.4 | 45.9 | 70.2 | 56.1 | 43.2 | 41.8 | 57.1 | — | |
| Pre-trained2026.03 | 80.34 | 79.54 | 50.72 | 78.37 | 73.72 | 79.76 | 49.32 | 45.2 | 67.12 | — | |
| FedEx-LoRA2026.03 | 80.02 | 77.4 | 51.9 | 72.3 | 72.57 | 79.55 | 47.99 | 47.9 | 66.2 | — | |
| FRAMEBackbone=Qwen2.5-7B, Param %=0.312026.06 | 77.3 | 91.1 | 83.5 | 96.5 | 87.4 | 94.2 | 85 | 90.4 | 88.2 | — | |
| MoABackbone=Qwen2.5-7B, Param %=0.402026.06 | 76.6 | 90.5 | 82.8 | 96.1 | 86.5 | 93.6 | 84 | 89.6 | 87.5 | — | |
| FLoRA2026.03 | 76.53 | 77.15 | 48.72 | 73.54 | 70.72 | 76.64 | 43.43 | 42.1 | 63.6 | — | |
| FourierMoEBackbone=Qwen2.5-7B, Param %=0.432026.06 | 76.5 | 90.4 | 82.7 | 96 | 86.4 | 93.5 | 83.9 | 89.5 | 87.4 | — | |
| Full FTPEFT=Full FT, r/N=N/A, #Params=100%, Backbone=Llama3-8B, Precision=FP162025.06 | 76.4 | 89.7 | 82.5 | 95.5 | 89.6 | 92.9 | 84.3 | 89.2 | 87.5 | — | |
| FlyLoRABackbone=Qwen2.5-7B, Param %=0.132026.06 | 76.4 | 90.3 | 82.6 | 95.9 | 86.3 | 93.4 | 83.8 | 89.4 | 87.3 | — | |
| HMoRABackbone=Qwen2.5-7B, Param %=0.972026.06 | 76.3 | 90.2 | 82.4 | 95.9 | 86.1 | 93.3 | 83.6 | 89.3 | 87.1 | — | |
| DiaBloPEFT=DiaBlo, r/N=N =128, #Params=0.52%, Backbone=Llama3-8B, Precision=FP162025.06 | 76.1 | 90.3 | 81.5 | 95.7 | 89.7 | 93.4 | 83.4 | 87.2 | 87.2 | — | |
| MixLoRABackbone=Qwen2.5-7B, Param %=1.292026.06 | 76 | 89.9 | 82.1 | 95.7 | 85.8 | 93 | 83.2 | 89 | 86.8 | — | |
| DiaBloPEFT=DiaBlo, r/N=N =64, #Params=1.04%, Backbone=Llama3-8B, Precision=FP162025.06 | 75.8 | 90.8 | 80.7 | 95.6 | 89.9 | 93.4 | 83 | 89.2 | 87.3 | — | |
| SMTPEFT=SMT∗, r/N=N/A, #Params=0.71%, Backbone=Llama3-8B, Precision=FP162025.06 | 75.7 | 88.4 | 81.4 | 96.2 | 88.2 | 92.7 | 83.2 | 88.6 | 86.8 | — | |
| DiaBloPEFT=DiaBlo, r/N=N = 128, #Params=0.53%, Backbone=Llama-13B, Precision=FP162025.06 | 75.7 | 84.8 | 82 | 90.6 | 86.8 | 90.1 | 77.1 | 89.2 | 84.5 | — | |
| LIFTModel=LLaMA-3-8B, Best Rank=322025.06 | 75.7 | 90.5 | 83.2 | 96.5 | 89.4 | 93.6 | 83.9 | 90.2 | 87.88 | — | |
| HydraLoRABackbone=Qwen2.5-7B, Param %=0.732026.06 | 75.7 | 89.6 | 81.8 | 95.5 | 85.5 | 92.8 | 82.9 | 88.7 | 86.6 | — | |
| HiRAModel=Llama3-8B, Params(%)=0.702026.03 | 75.4 | 89.7 | 81.2 | 95.4 | 87.7 | 93.3 | 82.9 | 88.3 | 86.7 | — | |
| Full FTModel=LLaMA-3-8B, Best Rank=–2025.06 | 75.4 | 88 | 81.8 | 96.5 | 89.3 | 93.1 | 83 | 86 | 86.64 | — | |
| FRAMEBackbone=LLaMA-3.1-8B, Param %=0.312026.06 | 75.4 | 90.3 | 82.8 | 95.9 | 87.8 | 92.8 | 83.4 | 88.5 | 87.1 | — | |
| DoRABackbone=Qwen2.5-7B, Param %=0.862026.06 | 75.4 | 89.3 | 81.4 | 95.3 | 85.1 | 92.5 | 82.4 | 88.3 | 86.2 | — | |
| FourierFTBackbone=Qwen2.5-7B, Param %=0.0222026.06 | 75.2 | 89.2 | 81.3 | 95.2 | 85 | 92.4 | 82.3 | 88.2 | 86.1 | — | |
| SMT(best)PEFT=SMT(best)∗, r/N=N/A, #Params=3.01%, Backbone=Llama3-8B, Precision=FP162025.06 | 75.1 | 89.9 | 82.4 | 96.3 | 88.8 | 92.6 | 82.8 | 89.6 | 87.2 | — | |
| AdaLoRABackbone=Qwen2.5-7B, Param %=0.852026.06 | 75.1 | 89.1 | 81.2 | 95.1 | 84.9 | 92.3 | 82.1 | 88.1 | 86 | — | |
| LoRABackbone=Qwen2.5-7B, Param %=0.852026.06 | 75 | 89 | 81 | 95 | 84.8 | 92.2 | 82 | 88 | 85.9 | — | |
| HydraLoRAModel Scale=Larger model scale2025.09 | 74.89 | 92.76 | 82.62 | 95.5 | 89.24 | 97.43 | 92.19 | 91.41 | 89.5 | — | |
| ALoRAModel Scale=Larger model scale2025.09 | 74.86 | 92.82 | 82.75 | 95.74 | 89.15 | 97.58 | 92.24 | 91.6 | 89.59 | — | |
| DiaBloPEFT=DiaBlo, r/N=N =128, #Params=0.52%, Backbone=Llama2-7B, Precision=FP162025.06 | 74.8 | 85.5 | 80.9 | 89.9 | 85.7 | 88.5 | 76.3 | 86 | 83.5 | — | |
| Full FTPEFT=Full FT, r/N=N/A, #Params=100%, Backbone=Llama-13B, Precision=FP162025.06 | 74.8 | 86.2 | 80.5 | 90.5 | 87.1 | 88.6 | 75.4 | 88 | 83.9 | — | |
| LIFTModel=LLaMA-2-7B, Best Rank=1282025.06 | 74.8 | 84.7 | 82.2 | 94.4 | 86 | 89.2 | 76.4 | 89.6 | 84.66 | — | |
| DiaBloPEFT=DiaBlo, r/N=N = 64, #Params=1.06%, Backbone=Llama-13B, Precision=FP162025.06 | 74.7 | 86.9 | 81.2 | 91.5 | 87.4 | 89.8 | 78.2 | 90.2 | 84.9 | — | |
| DoRA# Params (%)=0.71%, Base Model=LLaMA3 8 billion parameter model2024.05 | 74.6 | 89.3 | 79.9 | 95.5 | 85.6 | 90.5 | 80.4 | 85.8 | 85.2 | — | |
| DoRAModel=LLaMA3-8B, #Params.=0.71%2024.08 | 74.6 | 89.3 | 79.9 | 95.5 | 85.6 | 90.5 | 80.4 | 85.8 | 85.2 | — | |
| RoAd2Model=LLaMA3-8B, #Params.=0.03%2024.08 | 74.6 | 89.8 | 81.6 | 96 | 86.9 | 92.8 | 82.1 | 86.8 | 86.3 | — | |
| DoRAPEFT=DoRA†, r/N=r =32, #Params=0.78%, Backbone=Llama3-8B, Precision=FP162025.06 | 74.6 | 89.3 | 79.9 | 95.5 | 85.6 | 90.5 | 80.4 | 85.8 | 85.2 | — | |
| DoRAModel=Llama3-8B, Params(%)=0.712026.03 | 74.6 | 89.3 | 79.9 | 95.5 | 85.6 | 90.5 | 80.4 | 85.8 | 85.2 | — | |
| DoRAModel=LLaMA-3-8B, Best Rank=642025.06 | 74.6 | 87.4 | 81.2 | 94.7 | 87.1 | 89.4 | 79.5 | 86.4 | 85.04 | — | |
| MoABackbone=LLaMA-3.1-8B, Param %=0.392026.06 | 74.6 | 89.7 | 82 | 95.5 | 86.9 | 92.1 | 82.2 | 87.6 | 86.3 | — | |
| DoRA# Params (%)=0.35%, Base Model=LLaMA3 8 billion parameter model2024.05 | 74.5 | 88.8 | 80.3 | 95.5 | 84.7 | 90.1 | 79.1 | 87.2 | 85 | — | |
| DoRAModel=LLaMA3-8B, #Params.=0.35%2024.08 | 74.5 | 88.8 | 80.3 | 95.5 | 84.7 | 90.1 | 79.1 | 87.2 | 85 | — | |
| DoRAPEFT=DoRA†, r/N=r =16, #Params=0.39%, Backbone=Llama3-8B, Precision=FP162025.06 | 74.5 | 88.8 | 80.3 | 95.5 | 84.7 | 90.1 | 79.1 | 87.2 | 85 | — | |
| DoRAhalfModel=Llama3-8B, Params(%)=0.362026.03 | 74.5 | 88.8 | 80.3 | 95.5 | 84.7 | 90.1 | 79.1 | 87.2 | 85 | — | |
| FourierMoEBackbone=LLaMA-3.1-8B, Param %=0.422026.06 | 74.5 | 89.6 | 81.9 | 95.4 | 86.8 | 92 | 82 | 87.5 | 86.2 | — | |
| MTL-LoRABackbone=LLaMA3-8B2025.09 | 74.47 | 87.95 | 80.67 | 94.88 | 85.72 | 89.5 | 79.69 | 85.81 | 84.84 | -0.001 | |
| RoAd4Model=LLaMA3-8B, #Params.=0.07%2024.08 | 74.4 | 89.8 | 81.1 | 96.2 | 87.8 | 92.9 | 83 | 86.8 | 86.5 | — | |
| FlyLoRABackbone=LLaMA-3.1-8B, Param %=0.132026.06 | 74.4 | 89.5 | 81.8 | 95.3 | 86.7 | 91.9 | 81.9 | 87.4 | 86.1 | — | |
| ALoRABackbone=LLaMA3-8B2025.09 | 74.38 | 88.24 | 80.17 | 94.85 | 85.68 | 89.69 | 79.4 | 86.1 | 84.81 | 0.04 | |
| LoRETTA# Params (%)=0.13%, Base Model=LLaMA3 8 billion parameter model2024.05 | 74.3 | 87.5 | 80.9 | 94.5 | 86.7 | 92.1 | 81.5 | 85.8 | 85.4 | — | |
| QuanTA# Params (%)=0.035%, Base Model=LLaMA3 8 billion parameter model2024.05 | 74.3 | 88.1 | 81.8 | 95.1 | 87.3 | 91.1 | 81.7 | 87.2 | 85.8 | — | |
| HMoRABackbone=LLaMA-3.1-8B, Param %=0.952026.06 | 74.2 | 89.4 | 81.6 | 95.2 | 86.5 | 91.8 | 81.6 | 87.2 | 85.9 | — | |
| CoLABackbone=LLaMA3-8B2025.09 | 74.17 | 87.48 | 79.61 | 94.78 | 85.06 | 89.09 | 79.3 | 85.19 | 84.34 | 0.6 | |
| DiaBloPEFT=DiaBlo, r/N=N =64, #Params=1.04%, Backbone=Llama2-7B, Precision=FP162025.06 | 73.9 | 84.8 | 81.7 | 90 | 85 | 87.9 | 76.8 | 86.8 | 83.4 | — | |
| MixLoRABackbone=LLaMA-3.1-8B, Param %=1.272026.06 | 73.9 | 89.1 | 81.3 | 95 | 86.2 | 91.5 | 81.2 | 86.8 | 85.6 | — | |
| Full FTModel=LLaMA-2-7B, Best Rank=–2025.06 | 73.8 | 84.2 | 81 | 94.7 | 85.2 | 88.9 | 75.6 | 84.8 | 83.53 | — | |
| CRISPModel=Llama3-8B, Params(%)=0.012026.03 | 73.6 | 89.1 | 80.8 | 94.8 | 85.7 | 93.1 | 83 | 87.6 | 86 | — | |
| HydraLoRABackbone=LLaMA-3.1-8B, Param %=0.712026.06 | 73.6 | 88.8 | 81 | 94.8 | 85.9 | 91.2 | 80.8 | 86.4 | 85.3 | — | |
| RoAd1Model=LLaMA3-8B, #Params.=0.02%2024.08 | 73.5 | 89 | 81.4 | 96 | 87.6 | 92.9 | 82.4 | 87.2 | 86.3 | — | |
| DoRABackbone=LLaMA-3.1-8B, Param %=0.842026.06 | 73.4 | 88.5 | 80.6 | 94.6 | 85.4 | 90.9 | 80.3 | 85.9 | 85 | — | |
| ST BaselineBackbone=LLaMA3-8B2025.09 | 73.39 | 89.25 | 80.69 | 95.4 | 85.35 | 90.81 | 77.76 | 86.6 | 84.9 | — | |
| LoRAMoEBackbone=LLaMA3-8B2025.09 | 73.34 | 86.24 | 80.31 | 93.97 | 83.57 | 88.01 | 76.53 | 83.8 | 83.22 | 1.92 | |
| Full FTPEFT=Full FT, r/N=N/A, #Params=100%, Backbone=Llama2-7B, Precision=FP162025.06 | 73.3 | 85.7 | 81 | 90.2 | 86.9 | 88.6 | 77.4 | 85.2 | 83.5 | — | |
| S2FTModel=LLaMA-2-7B, Best Rank=1282025.06 | 73.3 | 83.7 | 81 | 94.3 | 84.6 | 88.3 | 75.8 | 84.8 | 83.22 | — | |
| ChatGPTModel=ChatGPT2026.03 | 73.1 | 85.4 | 68.5 | 78.5 | 66.1 | 89.8 | 79.9 | 74.8 | 77 | — | |
| FourierFTBackbone=LLaMA-3.1-8B, Param %=0.0212026.06 | 73.1 | 88.4 | 80.4 | 94.5 | 85.3 | 90.8 | 80.1 | 85.7 | 84.8 | — | |
| RoAd2Model=LLaMA2-7B, #Params.=0.04%2024.08 | 73 | 83.9 | 80.2 | 93.2 | 83 | 86.5 | 74.4 | 83 | 82.2 | — | |
| KronA# Params (%)=0.052%, Base Model=LLaMA3 8 billion parameter model2024.05 | 72.9 | 87.1 | 80.6 | 92.1 | 85.1 | 87.8 | 76 | 84.3 | 83.2 | — | |
| AdaLoRABackbone=LLaMA-3.1-8B, Param %=0.832026.06 | 72.9 | 88.3 | 80.1 | 94.4 | 85 | 90.7 | 79.9 | 85.6 | 84.6 | — | |
| TLoRATrainable Parameters=41.68M2026.04 | 72.87 | 86.28 | 82.59 | 95.21 | 86.58 | 88.59 | 76.36 | 85.2 | 84.21 | — | |
| PiCar=128, #Params=5.11M2025.05 | 72.84 | 87.98 | 77.79 | 92.82 | 79.4 | 93.14 | 83.62 | 88.2 | 84.47 | — | |
| LoRABackbone=LLaMA-3.1-8B, Param %=0.832026.06 | 72.8 | 88.1 | 80.2 | 94.3 | 85.1 | 90.6 | 79.8 | 85.4 | 84.5 | — | |
| DoRATrainable Parameters=81.31M2026.04 | 72.62 | 85.03 | 81.52 | 94.81 | 85.79 | 88.29 | 75.42 | 85.4 | 83.61 | — | |
| RoAd4Model=LLaMA2-7B, #Params.=0.08%2024.08 | 72.6 | 83.8 | 80 | 93.3 | 83 | 87.1 | 73.7 | 84.8 | 82.3 | — | |
| SMT(Best)PEFT=SMT(Best)∗, r/N=N/A, #Params=4.91%, Backbone=Llama2-7B, Precision=FP162025.06 | 72.6 | 85.2 | 82 | 94.4 | 85.7 | 87.8 | 74.5 | 85 | 83.4 | — | |
| DoRAPEFT=DoRA, r/N=r = 32, #Params=0.68%, Backbone=Llama-13B, Precision=FP162025.06 | 72.5 | 85.3 | 79.9 | 90.1 | 82.9 | 82.7 | 69.7 | 83.6 | 80.8 | — | |
| PiSSAModel=LLaMA-2-7B, Best Rank=1282025.06 | 72.5 | 85.3 | 80.8 | 87.2 | 86.1 | 87.1 | 74.3 | 85.6 | 82.36 | — |