Multitask Language Understanding on MMLU (val)
74.12AccuracyGWT
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| GWTModel=Qwen2.5-7B2025.01 | 74.12 | 70.38 | 83.88 | 68.06 | 77.14 | |
| LoRAModel=Qwen2.5-7B2025.01 | 73.85 | 70.44 | 83.23 | 67.8 | 76.9 | |
| APOLLOModel=Qwen2.5-7B2025.01 | 73.77 | 70.51 | 83.43 | 67.35 | 76.96 | |
| GaLoreModel=Qwen2.5-7B2025.01 | 73.76 | 70.21 | 83.49 | 67.52 | 76.87 | |
| Model SwarmBackbone=Qwen2.5-1.5B2026.05 | 64 | — | — | — | — | |
| EvoGMBackbone=Qwen2.5-1.5B2026.05 | 64 | — | — | — | — | |
| PSO-MergingBackbone=Qwen2.5-1.5B2026.05 | 63.5 | — | — | — | — | |
| TAIATraining Dataset=CoT-Collection, Model=LLaMA3-8B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 63.16 | — | — | — | — | |
| MTLBase model=Qwen2.5-1.5B2026.05 | 62 | — | — | — | — | |
| Task ArithmeticBase model=Qwen2.5-1.5B2026.05 | 62 | — | — | — | — | |
| EvoGMBase model=Qwen2.5-1.5B2026.05 | 62 | — | — | — | — | |
| MTLBackbone=Qwen2.5-1.5B2026.05 | 62 | — | — | — | — | |
| TABackbone=Qwen2.5-1.5B2026.05 | 62 | — | — | — | — | |
| TAIATraining Dataset=Alpaca-GPT4, Model=LLaMA3-8B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 61.87 | — | — | — | — | |
| TAIATraining Dataset=CoT-Collection, Model=LLaMA3-8B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 61.82 | — | — | — | — | |
| MOLORATraining Dataset=Alpaca-GPT4, Model=LLaMA3-8B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 61.78 | — | — | — | — | |
| Single BestBase model=Qwen2.5-1.5B2026.05 | 61.5 | — | — | — | — | |
| Single BestBackbone=Qwen2.5-1.5B2026.05 | 61.5 | — | — | — | — | |
| CMABackbone=Qwen2.5-1.5B2026.05 | 61.5 | — | — | — | — | |
| TAIATraining Dataset=Alpaca-GPT4, Model=LLaMA3-8B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 61.42 | — | — | — | — | |
| Model SoupBase model=Qwen2.5-1.5B2026.05 | 61 | — | — | — | — | |
| Model SwarmBase model=Qwen2.5-1.5B2026.05 | 61 | — | — | — | — | |
| Model SoupBackbone=Qwen2.5-1.5B2026.05 | 61 | — | — | — | — | |
| LoRATraining Dataset=Alpaca-GPT4, Model=LLaMA3-8B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 60.84 | — | — | — | — | |
| LoRATraining Dataset=CoT-Collection, Model=LLaMA3-8B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 60.6 | — | — | — | — | |
| DAREBase model=Qwen2.5-1.5B2026.05 | 60.5 | — | — | — | — | |
| DAREBackbone=Qwen2.5-1.5B2026.05 | 60.5 | — | — | — | — | |
| MOLORATraining Dataset=CoT-Collection, Model=LLaMA3-8B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 60.07 | — | — | — | — | |
| TIESBase model=Qwen2.5-1.5B2026.05 | 60 | — | — | — | — | |
| CMABase model=Qwen2.5-1.5B2026.05 | 60 | — | — | — | — | |
| TIESBackbone=Qwen2.5-1.5B2026.05 | 60 | — | — | — | — | |
| AdamModel=LLaMA3.2-3B2025.01 | 59.4 | 49.5 | 68.25 | 56.28 | 64.74 | |
| LLaMA3-8BTraining Dataset=Base Model, Model=LLaMA3-8B2024.05 | 59.38 | — | — | — | — | |
| GaLoreModel=LLaMA3.2-3B2025.01 | 59.37 | 49.73 | 68.7 | 55.94 | 64.47 | |
| GWTModel=LLaMA3.2-3B2025.01 | 59.34 | 49.5 | 68.61 | 56.56 | 64.62 | |
| APOLLOModel=LLaMA3.2-3B2025.01 | 59.32 | 49.17 | 68.18 | 56.32 | 64.71 | |
| LoRAModel=LLaMA3.2-3B2025.01 | 59.23 | 49.83 | 67.63 | 56.56 | 63.88 | |
| PSO-MergingBase model=Qwen2.5-1.5B2026.05 | 58 | — | — | — | — | |
| Qwen1.5-7BTraining Dataset=Base Model, Model=Qwen1.5-7B2024.05 | 57.69 | — | — | — | — | |
| TAIATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-7B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 57.29 | — | — | — | — | |
| TAIATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-7B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 57.15 | — | — | — | — | |
| TAIATraining Dataset=CoT-Collection, Model=Qwen1.5-7B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 57.01 | — | — | — | — | |
| BaseBase model=Qwen2.5-1.5B2026.05 | 57 | — | — | — | — | |
| BaseBackbone=Qwen2.5-1.5B2026.05 | 57 | — | — | — | — | |
| TAIATraining Dataset=CoT-Collection, Model=Qwen1.5-7B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 56.52 | — | — | — | — | |
| MOLORATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-7B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 55.03 | — | — | — | — | |
| LoRATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-7B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 54.89 | — | — | — | — | |
| GWTModel=Gemma3-4B2025.01 | 54.56 | 46.39 | 64.87 | 49.73 | 59.38 | |
| AdamModel=Gemma3-4B2025.01 | 54.46 | 46.59 | 64.74 | 49.71 | 58.91 | |
| GaLoreModel=Gemma3-4B2025.01 | 54.26 | 46.36 | 64.51 | 49.42 | 58.91 | |
| APOLLOModel=Gemma3-4B2025.01 | 54.17 | 46.49 | 64.28 | 49.39 | 58.67 | |
| LoRAModel=Gemma3-4B2025.01 | 53.41 | 45.36 | 63.41 | 48.63 | 58.36 | |
| BaselineBackbone=Llama-2-13b, Param Ratio=1.00, Fine-tuning=None2025.12 | 52.1 | — | — | — | — | |
| LLRCBackbone=Llama-2-13b, Param Ratio=0.90, Fine-tuning=None2025.12 | 49.7 | — | — | — | — | |
| LLM-Pruner+FinetuneBackbone=Llama-2-13b, Param Ratio=0.90, Fine-tuning=Alpaca2025.12 | 48 | — | — | — | — | |
| LoRATraining Dataset=CoT-Collection, Model=LLaMA2-7B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 46.99 | — | — | — | — | |
| LLM-PrunerBackbone=Llama-2-13b, Param Ratio=0.90, Fine-tuning=None2025.12 | 45.4 | — | — | — | — | |
| TAIATraining Dataset=CoT-Collection, Model=LLaMA2-7B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 45.34 | — | — | — | — | |
| SVD-LLM+FinetuneBackbone=Llama-2-13b, Param Ratio=0.90, Fine-tuning=Alpaca2025.12 | 45.1 | — | — | — | — | |
| MOLORATraining Dataset=CoT-Collection, Model=LLaMA2-7B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 45.09 | — | — | — | — | |
| TAIATraining Dataset=CoT-Collection, Model=Qwen1.5-1.8B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 44.13 | — | — | — | — | |
| LLRCBackbone=Llama-2-13b, Param Ratio=0.80, Fine-tuning=None2025.12 | 44.1 | — | — | — | — | |
| TAIATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-1.8B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 43.73 | — | — | — | — | |
| Qwen1.5-1.8BTraining Dataset=Base Model, Model=Qwen1.5-1.8B2024.05 | 43.62 | — | — | — | — | |
| TAIATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-1.8B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 43.48 | — | — | — | — | |
| LoRATraining Dataset=Alpaca-GPT4, Model=LLaMA2-7B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 43.29 | — | — | — | — | |
| MOLORATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-1.8B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 42.58 | — | — | — | — | |
| TAIATraining Dataset=CoT-Collection, Model=LLaMA2-7B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 42.54 | — | — | — | — | |
| TAIATraining Dataset=CoT-Collection, Model=Qwen1.5-1.8B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 42.39 | — | — | — | — | |
| LLaMA2-7BTraining Dataset=Base Model, Model=LLaMA2-7B2024.05 | 42.3 | — | — | — | — | |
| TAIATraining Dataset=Alpaca-GPT4, Model=LLaMA2-7B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 41.91 | — | — | — | — | |
| LoRATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-1.8B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 41.67 | — | — | — | — | |
| BaselineBackbone=Llama-2-7b, Param Ratio=1.00, Fine-tuning=None2025.12 | 41.3 | — | — | — | — | |
| SVD-LLM+FinetuneBackbone=Llama-2-13b, Param Ratio=0.80, Fine-tuning=Alpaca2025.12 | 41.3 | — | — | — | — | |
| TAIATraining Dataset=Alpaca-GPT4, Model=LLaMA2-7B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 41.21 | — | — | — | — | |
| MOLORATraining Dataset=Alpaca-GPT4, Model=LLaMA2-7B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 41.18 | — | — | — | — | |
| LLRCBackbone=Llama-2-7b, Param Ratio=0.90, Fine-tuning=None2025.12 | 37.8 | — | — | — | — | |
| LLM-Pruner+FinetuneBackbone=Llama-2-7b, Param Ratio=0.90, Fine-tuning=Alpaca2025.12 | 34 | — | — | — | — | |
| SVD-LLM+FinetuneBackbone=Llama-2-7b, Param Ratio=0.90, Fine-tuning=Alpaca2025.12 | 33 | — | — | — | — | |
| LLM-Pruner+FinetuneBackbone=Llama-2-13b, Param Ratio=0.80, Fine-tuning=Alpaca2025.12 | 32.1 | — | — | — | — | |
| SVD-LLM (w)Backbone=Llama-2-13b, Param Ratio=0.90, Fine-tuning=None2025.12 | 31.6 | — | — | — | — | |
| SVD-LLM (w)Backbone=Llama-2-13b, Param Ratio=0.80, Fine-tuning=None2025.12 | 29.6 | — | — | — | — | |
| LLRCBackbone=Llama-2-7b, Param Ratio=0.80, Fine-tuning=None2025.12 | 29.3 | — | — | — | — | |
| LLM-PrunerBackbone=Llama-2-7b, Param Ratio=0.90, Fine-tuning=None2025.12 | 28.5 | — | — | — | — | |
| SVD-LLM+FinetuneBackbone=Llama-2-7b, Param Ratio=0.80, Fine-tuning=Alpaca2025.12 | 27.1 | — | — | — | — | |
| LLM-Pruner+FinetuneBackbone=Llama-2-7b, Param Ratio=0.80, Fine-tuning=Alpaca2025.12 | 26.7 | — | — | — | — | |
| LoRATraining Dataset=CoT-Collection, Model=Qwen1.5-1.8B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 25.51 | — | — | — | — | |
| MOLORATraining Dataset=CoT-Collection, Model=Qwen1.5-7B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 25.37 | — | — | — | — | |
| LLM-PrunerBackbone=Llama-2-7b, Param Ratio=0.80, Fine-tuning=None2025.12 | 25.1 | — | — | — | — | |
| LoRATraining Dataset=CoT-Collection, Model=Qwen1.5-7B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 25 | — | — | — | — | |
| SVD-LLM (w)Backbone=Llama-2-7b, Param Ratio=0.90, Fine-tuning=None2025.12 | 24 | — | — | — | — | |
| MOLORATraining Dataset=CoT-Collection, Model=Qwen1.5-1.8B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 23.88 | — | — | — | — | |
| SVD-LLM (w)Backbone=Llama-2-7b, Param Ratio=0.80, Fine-tuning=None2025.12 | 23.8 | — | — | — | — | |
| LLM-PrunerBackbone=Llama-2-13b, Param Ratio=0.80, Fine-tuning=None2025.12 | 23.2 | — | — | — | — |