Commonsense Reasoning Suite Evaluation
93Avg ScoreDual LoRA
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dual LoRAModel=LLaMA3-70B-Instruct, Rank (r)=8, Trainable Param (%)=0.1972025.12 | 93 | 82.5 | 95.2 | 85.8 | 97.7 | 94.6 | 98.2 | 93.6 | 96.6 | |
| DoRAModel=LLaMA3-70B-Instruct, Rank (r)=16, Trainable Param (%)=0.2022025.12 | 91.1 | 78.4 | 93 | 83.8 | 96.5 | 92.3 | 97.6 | 92.3 | 94.6 | |
| LoFTBackbone=LLaMA-3-70B, Rank (r)=162025.05 | 90.66 | 79.45 | 92.82 | 82.24 | 97.02 | 90.06 | 97.52 | 92.75 | 93.4 | |
| LoRAModel=LLaMA3-70B-Instruct, Rank (r)=16, Trainable Param (%)=0.1972025.12 | 90.5 | 78.6 | 92.8 | 83.4 | 92.7 | 92.6 | 97.5 | 91.7 | 94.4 | |
| LoFTBackbone=LLaMA-3-70B, Rank (r)=42025.05 | 90.17 | 79.17 | 92.6 | 81.58 | 96.7 | 89.34 | 97.31 | 92.06 | 92.6 | |
| LoFTBackbone=LLaMA-3-70B, Rank (r)=22025.05 | 89.94 | 78.38 | 92.55 | 81.78 | 96.61 | 88.4 | 97.31 | 92.06 | 92.4 | |
| TalkLoRALLM=Qwen2.5-7B, rank (r)=16, #Param(%)=0.22026.04 | 89 | 73.6 | 90.9 | 83 | 95.5 | 89.9 | 96.5 | 89.6 | 92.8 | |
| LoFTBackbone=LLaMA-3-70B, Rank (r)=12025.05 | 88.93 | 77.83 | 91.51 | 80.35 | 95.79 | 86.03 | 97.05 | 91.47 | 91.4 | |
| Qwen2.5-7B+MASAttention=MAS2024.12 | 88.8 | 73.7 | 90.3 | 83.5 | 94.5 | 88.8 | 96.7 | 90.1 | 93 | |
| TeamLoRALLM=Qwen2.5-7B, rank (r)=16, #Param(%)=0.4, Reproduced=true2026.04 | 88.5 | 74.6 | 90 | 82.3 | 95.4 | 89 | 95.9 | 88.5 | 92.2 | |
| DiaBloBackbone=Llama3-8B, r/N=N = 64, #Params=1.51%, Precision=BF162025.06 | 88.3 | 76.6 | 90.2 | 83.1 | 96.9 | 88.8 | 93.9 | 85.1 | 91.5 | |
| Full FTBackbone=Mistral-7B, r/N=NA, #Params=100%, Precision=BF162025.06 | 88.2 | 76.6 | 90.6 | 83.5 | 97 | 90 | 93.4 | 82.9 | 91.5 | |
| DiaBloBackbone=Mistral-7B, r/N=N = 64, #Params=1.68%, Precision=BF162025.06 | 88 | 76.6 | 90.9 | 83.7 | 96.9 | 88.6 | 93.8 | 83.2 | 90.5 | |
| DiaBloBackbone=Llama3-8B, r/N=N = 128, #Params=0.76%, Precision=BF162025.06 | 88 | 76.2 | 90.5 | 83.7 | 97 | 89 | 94 | 84.7 | 88.9 | |
| DiaBloBackbone=Mistral-7B, r/N=N = 128, #Params=0.84%, Precision=BF162025.06 | 87.9 | 76.7 | 91.3 | 83.9 | 96.9 | 88.2 | 93.1 | 83.4 | 89.7 | |
| Full FTBackbone=Llama3-8B, r/N=NA, #Params=100%, Precision=BF162025.06 | 87.9 | 76.2 | 90.3 | 84 | 96.8 | 88.3 | 93.3 | 84 | 90.5 | |
| TalkLoRALLM=LLaMA3-8B, rank (r)=32, #Param(%)=0.4, Number of Experts=82026.04 | 87.8 | 75.1 | 89.2 | 83.6 | 96.4 | 90.1 | 93.9 | 84.9 | 88.8 | |
| TalkLoRALLM=LLaMA3-8B, rank (r)=32, #Param(%)=0.42026.04 | 87.6 | 76.1 | 89.6 | 82.3 | 96 | 89.2 | 93.9 | 84.5 | 89.4 | |
| Dual LoRAModel=LLaMA3-8B, Rank (r)=16, Trainable Param (%)=0.72025.12 | 87.5 | 76.2 | 89.1 | 83.2 | 96 | 89.1 | 93.7 | 85.4 | 87.6 | |
| DoRABackbone=Mistral-7B, r/N=r = 32, #Params=1.25%, Precision=BF162025.06 | 87.5 | 75.8 | 90.4 | 82.9 | 96.3 | 87.9 | 92.6 | 83.3 | 90.6 | |
| TalkLoRALLM=LLaMA3-8B, rank (r)=16, #Param(%)=0.22026.04 | 87.4 | 75.3 | 88.8 | 82.3 | 96.2 | 89.6 | 93.2 | 84.3 | 89.2 | |
| SMoA(r=32,n=2)Model=Llama-3-8B, Params(%)=0.70022026.01 | 87.35 | 76.77 | 88.87 | 82.56 | 95.88 | 88.57 | 93.18 | 83.59 | 89.36 | |
| LoRIBackbone=Llama3-8B, r/N=r = 32, #Params=0.56%, Precision=BF162025.06 | 87.3 | 76.4 | 89 | 82.7 | 95.9 | 87.9 | 93.6 | 84.2 | 88.5 | |
| LoRIBackbone=Mistral-7B, r/N=r = 32, #Params=0.63%, Precision=BF162025.06 | 87.1 | 75.9 | 90.6 | 83 | 95.9 | 87.4 | 91.9 | 83.6 | 88.4 | |
| LoRABackbone=Llama3-8B, r/N=r = 32, #Params=1.12%, Precision=BF162025.06 | 87.1 | 76.3 | 89.8 | 82.7 | 95.8 | 88.7 | 91.7 | 83.4 | 88.4 | |
| DoRABackbone=Llama3-8B, r/N=r = 32, #Params=1.12%, Precision=BF162025.06 | 87.1 | 75.9 | 89.8 | 82.7 | 95.3 | 88.2 | 93.2 | 83.5 | 87.9 | |
| LoRABackbone=Mistral-7B, r/N=r = 32, #Params=1.25%, Precision=BF162025.06 | 86.9 | 75.2 | 90.1 | 82.9 | 95.1 | 88.1 | 92 | 82.9 | 88.7 | |
| HiRA(r=32)Model=Llama-3-8B, Params(%)=0.70022026.01 | 86.73 | 75.4 | 89.7 | 81.15 | 95.36 | 87.7 | 93.27 | 82.9 | 88.32 | |
| HiRAModel=LLaMA3-8B, Rank (r)=32, Trainable Param (%)=0.72025.12 | 86.7 | 75.4 | 89.7 | 81.2 | 95.4 | 87.7 | 93.3 | 82.9 | 88.3 | |
| HiRALLM=LLaMA3-8B, rank (r)=32, #Param(%)=0.72026.04 | 86.7 | 75.4 | 89.7 | 81.2 | 95.4 | 87.7 | 93.3 | 82.9 | 88.3 | |
| S2FTModel=LLaMA3-8B, Trainable Parameters=0.70%2024.12 | 86.6 | 75 | 89 | 80.7 | 96.5 | 88 | 92.5 | 83.4 | 87.8 | |
| Qwen2.5-7BAttention=Causal2024.12 | 86.6 | 72.2 | 90.1 | 79.4 | 94.4 | 83.3 | 95.2 | 87.6 | 90.6 | |
| MoELoRALLM=LLaMA3-8B, rank (r)=32, #Param(%)=0.7, Reproduced=true2026.04 | 86.6 | 74.6 | 89.1 | 82.3 | 95.3 | 88.5 | 92.7 | 82.8 | 87.6 | |
| TeamLoRALLM=LLaMA3-8B, rank (r)=32, #Param(%)=0.7, Reproduced=true2026.04 | 86.4 | 74.3 | 88.2 | 81.8 | 95.4 | 89 | 92.8 | 81.7 | 88 | |
| SMoA(r=16,n=2)Model=Llama-3-8B, Params(%)=0.35132026.01 | 86.25 | 74.35 | 88.25 | 81.37 | 95.13 | 87.33 | 92.38 | 82.92 | 88.26 | |
| HiRA(r=16)Model=Llama-3-8B, Params(%)=0.35132026.01 | 86.08 | 73.85 | 89.12 | 81.06 | 94.85 | 86.74 | 93.06 | 82.59 | 87.4 | |
| Llama-3-8B+MASAttention=MAS2024.12 | 85.8 | 74.9 | 88.6 | 81.5 | 93.2 | 88 | 92.3 | 81.5 | 86.8 | |
| RandLoRAModel=LLaMA3-8B, Trainable Param (%)=0.72025.12 | 85.6 | 76.3 | 88.1 | 80.3 | 95.7 | 86.1 | 90.4 | 80.9 | 87 | |
| Mistral-7B-v0.1+MASAttention=MAS2024.12 | 85.5 | 70.9 | 88.5 | 82.5 | 91.4 | 88.2 | 92.6 | 80.5 | 89.2 | |
| HiRALLM=Qwen2.5-7B, rank (r)=16, #Param(%)=0.4, Reproduced=true2026.04 | 85.4 | 69 | 88.3 | 80.8 | 92.3 | 81 | 95.4 | 88.7 | 88 | |
| DoRAModel=LLaMA3-8B, Trainable Parameters=0.71%2024.12 | 85.2 | 74.6 | 89.3 | 79.9 | 95.5 | 85.6 | 90.5 | 80.4 | 85.8 | |
| Llama-3.1-8B+MASAttention=MAS2024.12 | 85.2 | 74.8 | 88 | 82 | 91.6 | 86 | 92.2 | 81.7 | 85.8 | |
| DoRAModel=LLaMA3-8B, Rank (r)=32, Trainable Param (%)=0.712025.12 | 85.2 | 74.6 | 89.3 | 79.9 | 95.5 | 85.6 | 90.5 | 80.4 | 85.8 | |
| DoRA(r=32)Model=Llama-3-8B, Params(%)=0.70022026.01 | 85.2 | 74.6 | 89.3 | 79.9 | 95.5 | 85.6 | 90.5 | 80.4 | 85.8 | |
| DoRALLM=LLaMA3-8B, rank (r)=32, #Param(%)=0.72026.04 | 85.2 | 74.6 | 89.3 | 79.9 | 95.5 | 85.6 | 90.5 | 80.4 | 85.8 | |
| DoRALLM=LLaMA3-8B, Param(%)=0.71, Evaluation Protocol=Fine-tuning2025.05 | 85.2 | 74.6 | 89.3 | 79.9 | 95.5 | 85.6 | 90.5 | 80.4 | 85.8 | |
| Full FTModel=LLaMA-13B, Trainable Parameters=100%2024.12 | 85 | 74.5 | 86.3 | 81.3 | 94.4 | 86.9 | 89.7 | 77.9 | 88.8 | |
| DoRAModel=LLaMA3-8B, Rank (r)=16, Trainable Param (%)=0.352025.12 | 85 | 74.5 | 88.8 | 80.3 | 95.5 | 84.7 | 90.1 | 79.1 | 87.2 | |
| DoRALLM=LLaMA3-8B, Param(%)=0.35, Rank (r)=32, Evaluation Protocol=Fine-tuning2025.05 | 85 | 74.5 | 88.8 | 80.3 | 95.5 | 84.7 | 90.1 | 79.1 | 87.2 | |
| DenseLoRALLM=LLaMA3-8B, Param(%)=0.06, Rank (r)=64, Evaluation Protocol=Fine-tuning2025.05 | 85 | 74.1 | 88.9 | 80.3 | 95 | 87 | 90 | 79.2 | 85.6 | |
| MeLoRA(r=32,n=2)Model=Llama-3-8B, Params(%)=0.70022026.01 | 84.72 | 74.53 | 87.35 | 80.54 | 93.76 | 86.37 | 90.37 | 78.43 | 86.38 | |
| Mistral-7B-v0.1Attention=Causal2024.12 | 84.7 | 74.3 | 88.4 | 80 | 94.8 | 85.6 | 88.8 | 78.8 | 86.6 | |
| DenseLoRALLM=LLaMA3-8B, Param(%)=0.02, Rank (r)=32, Evaluation Protocol=Fine-tuning2025.05 | 84.6 | 74.3 | 88 | 80.3 | 94.5 | 86 | 89.7 | 78.7 | 85.6 | |
| S2FTModel=LLaMA-13B, Trainable Parameters=0.65%2024.12 | 84.3 | 74.2 | 85.7 | 80.7 | 94.9 | 86.4 | 88.4 | 76.3 | 87.8 | |
| Dual LoRAModel=LLaMA-13B, Rank (r)=16, Trainable Param (%)=0.672025.12 | 84.2 | 72.4 | 87.4 | 81.9 | 90.9 | 86.1 | 89.1 | 77.4 | 88.2 | |
| MixDoRABase Model=LLaMA-3 8B, Trainable Parameters=3.0%2024.04 | 84.1 | — | — | — | — | — | — | — | — | |
| Llama-3.1-8BAttention=Causal2024.12 | 84.1 | 73.5 | 87.5 | 80.3 | 94.4 | 85.3 | 88.8 | 78 | 84.8 | |
| Qwen2.5-3B+MASAttention=MAS2024.12 | 84.1 | 68.3 | 85.6 | 80.6 | 90.7 | 83.8 | 93.5 | 84 | 86.2 | |
| Llama-3-8BAttention=Causal2024.12 | 84 | 74.5 | 87.8 | 79.2 | 94.5 | 84.7 | 89.4 | 77.2 | 84.4 | |
| DenseLoRALLM=LLaMA3-8B, Param(%)=0.01, Rank (r)=16, Evaluation Protocol=Fine-tuning2025.05 | 83.8 | 72.3 | 87.5 | 79.8 | 93.5 | 85.2 | 89.8 | 78.2 | 84 | |
| CLoRAk=20482024.10 | 83.7 | 73.7 | 84.5 | 80.9 | 94.5 | 85.9 | 75.9 | 88.1 | 86 | |
| MeLoRA(r=16,n=2)Model=Llama-3-8B, Params(%)=0.35132026.01 | 83.67 | 72.85 | 87.33 | 79.32 | 92.78 | 84.38 | 88.65 | 78.38 | 85.63 | |
| Full FTModel=LLaMA3-8B, Trainable Parameters=100%2024.12 | 83.6 | 73.9 | 86.2 | 79.1 | 93.1 | 85.8 | 88.1 | 78.2 | 84 | |
| MixLoRABase Model=LLaMA-3 8B, Trainable Parameters=3.0%2024.04 | 83.5 | — | — | — | — | — | — | — | — | |
| MixLoRALLM=LLaMA3-8B, rank (r)=16, #Param(%)=3.02026.04 | 83.5 | 75 | 87.6 | 78.8 | 93.3 | 82.1 | 86.5 | 79.9 | 84.8 | |
| LoReFTModel=LLaMA-13B, Trainable Parameters=0.03%2024.12 | 83.3 | 72.1 | 86.3 | 81.8 | 95.1 | 87.2 | 86.2 | 73.7 | 84.2 | |
| MixDoRABase Model=LLaMA-2 13B, Trainable Parameters=2.5%2024.04 | 83.1 | — | — | — | — | — | — | — | — | |
| NoRALLM=LLaMA3-8B, Param(%)=0.10, Evaluation Protocol=Fine-tuning2025.05 | 83.1 | 73.3 | 86.4 | 79.1 | 94.1 | 84.3 | 88.2 | 77.5 | 85 | |
| MixLoRABase Model=LLaMA-2 13B, Trainable Parameters=2.5%2024.04 | 83 | — | — | — | — | — | — | — | — | |
| Full FTModel=LLaMA2-7B, Trainable Parameters=100%2024.12 | 83 | 74.7 | 84.9 | 78.7 | 93.7 | 84.1 | 87.5 | 75.2 | 85 | |
| S2FTModel=LLaMA2-7B, Trainable Parameters=0.81%2024.12 | 83 | 72.9 | 86.1 | 80.2 | 94.3 | 85.5 | 87.2 | 74.6 | 83.4 | |
| LoRAModel=LLaMA3-8B, Rank (r)=16, Trainable Param (%)=0.352025.12 | 83 | 71.7 | 86.8 | 79.5 | 93.9 | 84.4 | 87.4 | 76.3 | 84.2 | |
| TalkLoRALLM=LLaMA2-7B, rank (r)=32, #Param(%)=0.42026.04 | 82.9 | 73.1 | 84.8 | 80.9 | 89.2 | 86.5 | 87.9 | 75.7 | 84.8 | |
| CLoRAk=10242024.10 | 82.6 | 73.3 | 84.8 | 79.6 | 91.1 | 86.1 | 73.1 | 86.9 | 85.6 | |
| LoRAModel=LLaMA3-8B, Trainable Parameters=0.70%2024.12 | 82.5 | 70.8 | 85.2 | 79.7 | 92.5 | 84.9 | 88.9 | 78.7 | 84.4 | |
| TalkLoRALLM=LLaMA2-7B, rank (r)=16, #Param(%)=0.22026.04 | 82.5 | 72.6 | 83.9 | 81.5 | 89.5 | 85.2 | 88 | 73.5 | 86 | |
| Llama-3.2-3B+MASAttention=MAS2024.12 | 82.3 | 71.1 | 86 | 79.6 | 90.6 | 83.9 | 89.4 | 75.1 | 82.6 | |
| Dual LoRAModel=LLaMA2-7B, Rank (r)=16, Trainable Param (%)=0.832025.12 | 82.3 | 73.8 | 83.8 | 80.3 | 88.1 | 85.2 | 87.7 | 74.1 | 85 | |
| SMoA(r=32,n=2)Model=Llama-2-7B, Params(%)=0.82562026.01 | 82.08 | 72.86 | 83.86 | 81.12 | 89.04 | 84.65 | 87.38 | 74.35 | 83.38 | |
| CLoRAk=5122024.10 | 82 | 72.8 | 83 | 79.5 | 93 | 83.9 | 73 | 85.7 | 84.8 | |
| DoRABase Model=LLaMA-2 13B, Trainable Parameters=2.4%2024.04 | 81.9 | — | — | — | — | — | — | — | — | |
| Full FTModel=LLaMA-7B, Trainable Parameters=100%2024.12 | 81.9 | 70.3 | 84.2 | 80.1 | 92.3 | 85.4 | 86.6 | 72.8 | 83.4 | |
| S2FTModel=LLaMA-7B, Trainable Parameters=0.81%2024.12 | 81.8 | 72.7 | 83.7 | 79.6 | 93.4 | 83.5 | 86.1 | 72.2 | 83.4 | |
| LoRABase Model=LLaMA-2 13B, Trainable Parameters=2.4%2024.04 | 81.5 | — | — | — | — | — | — | — | — | |
| DoRAModel=LLaMA-13B, Trainable Parameters=0.68%2024.12 | 81.5 | 72.4 | 84.9 | 81.5 | 92.4 | 84.2 | 84.2 | 69.6 | 82.8 | |
| HiRA(r=32)Model=Llama-2-7B, Params(%)=0.82562026.01 | 81.42 | 71.22 | 83.35 | 79.53 | 88.12 | 83.98 | 86.74 | 73.81 | 84.6 | |
| ParallelModel=LLaMA-13B, Trainable Parameters=2.89%2024.12 | 81.4 | 72.5 | 84.9 | 79.8 | 92.1 | 84.7 | 84.2 | 71.2 | 82.4 | |
| HiRAModel=LLaMA2-7B, Rank (r)=32, Trainable Param (%)=0.832025.12 | 81.4 | 71.2 | 83.4 | 79.5 | 88.1 | 84 | 86.7 | 73.8 | 84.6 | |
| HiRALLM=LLaMA2-7B, rank (r)=32, #Param(%)=0.82026.04 | 81.4 | 71.2 | 83.4 | 79.5 | 88.1 | 84 | 86.7 | 73.8 | 84.6 | |
| AdaLoRALLM=LLaMA3-8B, Param(%)=0.35, Evaluation Protocol=Fine-tuning2025.05 | 81.4 | 75.1 | 86.4 | 76.7 | 75.4 | 83.3 | 90.4 | 79.1 | 81.4 | |
| DoRAModel=LLaMA-13B, Rank (r)=32, Trainable Param (%)=0.682025.12 | 81.1 | 72.4 | 84.9 | 81.2 | 91.5 | 83.7 | 84.6 | 68.9 | 81.6 | |
| LoRAModel=LLaMA3-8B, Rank (r)=32, Trainable Param (%)=0.72025.12 | 80.9 | 71.2 | 85.1 | 79.3 | 92.1 | 82.6 | 85.2 | 70.1 | 81.4 | |
| LoKrLLM=LLaMA3-8B, Param(%)=0.01, Evaluation Protocol=Fine-tuning2025.05 | 80.9 | 65.1 | 81.6 | 78.7 | 92 | 82.1 | 89.2 | 76.7 | 80.9 | |
| LoRAregularization=L22024.10 | 80.8 | 70.3 | 83 | 80.2 | 92.7 | 83.1 | 71.2 | 84.2 | 81.4 | |
| LoRALLM=LLaMA3-8B, rank (r)=32, #Param(%)=0.72026.04 | 80.8 | 70.8 | 85.2 | 79.9 | 91.7 | 84.3 | 84.2 | 71.2 | 79 | |
| LoRALLM=LLaMA3-8B, Param(%)=0.70, Evaluation Protocol=Fine-tuning2025.05 | 80.8 | 70.8 | 85.2 | 79.9 | 91.7 | 84.3 | 84.2 | 71.2 | 79 | |
| LoRA(r=32)Model=Llama-3-8B, Params(%)=0.70022026.01 | 80.79 | 70.8 | 85.2 | 79.9 | 91.7 | 84.3 | 84.2 | 71.2 | 79 | |
| CLoRAk=1282024.10 | 80.7 | 72.7 | 84.1 | 77.7 | 91.6 | 83 | 69.9 | 85.3 | 81.6 | |
| CLoRAk=2562024.10 | 80.7 | 71.3 | 83.2 | 79.1 | 92.4 | 83.2 | 71 | 84.5 | 81 | |
| Dual LoRAModel=LLaMA-7B, Rank (r)=16, Trainable Param (%)=0.832025.12 | 80.7 | 71.3 | 83.4 | 81 | 88.1 | 83.7 | 85 | 71 | 82.4 |