Mathematical Reasoning on GSM8K and MATH
78.4GSM8K ScoreC3A
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| C3ABackbone=LLaMA3-8B, Params (%)=0.26, Evaluation Protocol=Fine-tuned, Block size (b)=4096, Rank (r)=322024.07 | 78.4 | 29.9 | 54.1 | — | |
| LoRABackbone=LLaMA3-8B, Params (%)=0.70, Evaluation Protocol=Fine-tuned, Rank (r)=322024.07 | 77.2 | 28.2 | 52.7 | — | |
| DoRABackbone=LLaMA3-8B, Params (%)=0.71, Evaluation Protocol=Fine-tuned, Rank (r)=322024.07 | 77.2 | 30 | 53.6 | — | |
| VeRABackbone=LLaMA3-8B, Params (%)=0.04, Evaluation Protocol=Fine-tuned, Rank (r)=163842024.07 | 76 | 26.7 | 51.4 | — | |
| VB-LoRAModel=Gemma-7B, # Parameters=0.67M2024.05 | 75.96 | 28.9 | — | — | |
| LoRAModel=Gemma-7B, # Parameters=200M2024.05 | 74.9 | 31.28 | — | — | |
| LoRA-XSModel=Gemma-7B, # Parameters=0.80M2024.05 | 74.22 | 27.62 | — | — | |
| Full-FTModel=Gemma-7B, # Parameters=8538M2024.05 | 71.34 | 22.74 | — | — | |
| VB-LoRAModel=Mistral-7B, # Parameters=0.65M2024.05 | 69.22 | 17.9 | — | — | |
| LoRA-XSModel=Mistral-7B, # Parameters=0.92M2024.05 | 68.01 | 17.86 | — | — | |
| LoRAModel=Mistral-7B, # Parameters=168M2024.05 | 67.7 | 19.68 | — | — | |
| Full-FTModel=Mistral-7B, # Parameters=7242M2024.05 | 67.02 | 18.6 | — | — | |
| D3Paradigm=Ours, Backbone=Llama-3-8B, Evaluation Protocol=SFT2026.05 | 66.3 | 27.4 | 49.6 | — | |
| Loss low→highParadigm=Curriculum, Backbone=Llama-3-8B, Evaluation Protocol=SFT2026.05 | 64.3 | 24.1 | 46.2 | — | |
| DSL (Baseline)Paradigm=Training, Backbone=Llama-3-8B, Evaluation Protocol=SFT2026.05 | 63.8 | 24.6 | 46.5 | — | |
| DoRABackbone=LLaMA2-7B, Params (%)=0.84, Evaluation Protocol=Fine-tuned, Rank (r)=322024.07 | 62.3 | 12.4 | 37.3 | — | |
| MTLParadigm=Training, Backbone=Llama-3-8B, Evaluation Protocol=SFT2026.05 | 62.2 | 23.5 | 44.9 | — | |
| C3ABackbone=LLaMA2-7B, Params (%)=0.35, Evaluation Protocol=Fine-tuned, Block size (b)=4096, Rank (r)=322024.07 | 61.5 | 13 | 37.2 | — | |
| LoRABackbone=LLaMA2-7B, Params (%)=0.83, Evaluation Protocol=Fine-tuned, Rank (r)=322024.07 | 60.5 | 11.7 | 36.1 | — | |
| VeRABackbone=LLaMA2-7B, Params (%)=0.05, Evaluation Protocol=Fine-tuned, Rank (r)=163842024.07 | 58.5 | 9.4 | 34 | — | |
| MathBackbone=LLaMA2-13B2025.02 | 55.5 | 10.84 | — | — | |
| Sens-Merging (DARE)Backbone=LLaMA2-13B, Use Sens=true2025.02 | 55.42 | 9.08 | — | — | |
| DAREBackbone=LLaMA2-13B, Use Sens=false2025.02 | 55.19 | 9.08 | — | — | |
| Sens-Merging (Ties-Merging)Backbone=LLaMA2-13B, Use Sens=true2025.02 | 54.44 | 10.2 | — | — | |
| Sens-Merging (Task Arithmetic)Backbone=LLaMA2-13B, Use Sens=true2025.02 | 53.07 | 8.84 | — | — | |
| Ties-MergingBackbone=LLaMA2-13B, Use Sens=false2025.02 | 52.46 | 9.9 | — | — | |
| Task ArithmeticBackbone=LLaMA2-13B, Use Sens=false2025.02 | 49.89 | 7.32 | — | — | |
| FoLoRA#Param=320M2026.05 | 46.09 | 6.6 | 26.34 | 26.2 | |
| LoRA-Null#Param=320M2026.05 | 43.89 | 6.36 | 25.12 | 23.08 | |
| Loss high→lowParadigm=Curriculum, Backbone=Llama-3-8B, Evaluation Protocol=SFT2026.05 | 43.7 | 15.2 | 34.9 | — | |
| LoRA#Param=320M2026.05 | 42.92 | 6.09 | 24.5 | 20.85 | |
| MiLoRA#Param=320M2026.05 | 40.51 | 5.44 | 22.97 | 21.16 | |
| CorDA#Param=320M2026.05 | 40.26 | 5.65 | 22.95 | 21.45 | |
| Zero-ShotBackbone=LLaMA3-8B, Evaluation Protocol=Zero-Shot2024.07 | 33.1 | 5.3 | 19.2 | — | |
| ChatBackbone=LLaMA2-13B2025.02 | 32.37 | 6.7 | — | — | |
| OPLoRA#Param=320M2026.05 | 31.53 | 4.32 | 17.92 | 18.82 | |
| CodeBackbone=LLaMA2-13B2025.02 | 27.29 | 5.74 | — | — | |
| Zero-ShotBackbone=LLaMA2-7B, Evaluation Protocol=Zero-Shot2024.07 | 7.3 | 1.1 | 4.2 | — |