Language Understanding on MMLU (Subject Performance)
71.91Average AccuracyMASA
Evaluation results
| Method | Links | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MASABackbone=Qwen2.5-7B-Instruct, Rank (r)=16, #A=5, #B=12025.10 | 71.91 | — | 55.53 | — | 74.4 | — | — | — | — | — | — | — | — | — | — | — | 74.24 | |
| LoRABackbone=Qwen2.5-7B-Instruct, Rank (r)=16, #A=1, #B=12025.10 | 71.71 | — | 55.3 | — | 73.79 | — | — | — | — | — | — | — | — | — | — | — | 74.1 | |
| HydraLoRABackbone=Qwen2.5-7B-Instruct, Rank (r)=16, #A=1, #B=32025.10 | 71.47 | — | 54.74 | — | 72.7 | — | — | — | — | — | — | — | — | — | — | — | 72.03 | |
| CATModel Family=Llama3.1-8B2025.10 | 67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CAT*Model Family=Llama3.1-8B2025.10 | 67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| No DefenseModel Family=Llama3.1-8B2025.10 | 64 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Refusal TrainingModel Family=Llama3.1-8B2025.10 | 64 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAP TrainedModel Family=Llama3.1-8B2025.10 | 64 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LATModel Family=Llama3.1-8B2025.10 | 63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LAT*Model Family=Llama3.1-8B2025.10 | 63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| WildJailbreakModel Family=Llama3.1-8B2025.10 | 63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ASCoT (closed)Model Family=Llama3.1-8B2025.10 | 63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ASCoT (open)Model Family=Llama3.1-8B2025.10 | 63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CATModel Family=Mistral-7B2025.10 | 62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAP TrainedModel Family=Mistral-7B2025.10 | 61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MASABackbone=LLaMA3-8B, Zero-shot=true, Params. (%)=0.5220, A=5, B=12025.10 | 59.62 | — | — | — | — | — | — | — | — | — | — | — | — | 50.65 | 69.81 | 53.28 | 68.23 | |
| No DefenseModel Family=Mistral-7B2025.10 | 59 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LATModel Family=Mistral-7B2025.10 | 59 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LAT*Model Family=Mistral-7B2025.10 | 59 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ASCoT (closed)Model Family=Mistral-7B2025.10 | 59 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MASABackbone=Mistral-7B, Rank (r)=16, #A=5, #B=12025.10 | 58.99 | — | 48.33 | — | 64.71 | — | — | — | — | — | — | — | — | — | — | — | 59.96 | |
| LoRABackbone=Mistral-7B, Rank (r)=16, #A=1, #B=12025.10 | 58.83 | — | 47.93 | — | 64.03 | — | — | — | — | — | — | — | — | — | — | — | 59.9 | |
| LoRAr=16Backbone=LLaMA3-8B, Zero-shot=true, Params. (%)=0.5196, A=1, B=12025.10 | 58.54 | — | — | — | — | — | — | — | — | — | — | — | — | 50.46 | 68.05 | 52.41 | 66.59 | |
| DoRAr=16Backbone=LLaMA3-8B, Zero-shot=true, Params. (%)=0.5366, A=1, B=12025.10 | 58.37 | — | — | — | — | — | — | — | — | — | — | — | — | 49.92 | 68.22 | 52.26 | 66.46 | |
| HydraLoRABackbone=Mistral-7B, Rank (r)=16, #A=1, #B=32025.10 | 58.18 | — | 47.93 | — | 63.21 | — | — | — | — | — | — | — | — | — | — | — | 59.16 | |
| No DefenseModel Family=Zephyr-7B2025.10 | 58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CATModel Family=Zephyr-7B2025.10 | 58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LATModel Family=Zephyr-7B2025.10 | 58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LAT*Model Family=Zephyr-7B2025.10 | 58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Refusal TrainingModel Family=Mistral-7B2025.10 | 58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ASCoT (open)Model Family=Mistral-7B2025.10 | 58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VB-LoRABackbone=LLaMA3-8B, Zero-shot=true, Params. (%)=0.2613, A=1, B=12025.10 | 57.88 | — | — | — | — | — | — | — | — | — | — | — | — | 49.64 | 66.72 | 52.24 | 66.01 | |
| CoLABackbone=LLaMA3-8B, Zero-shot=true, Params. (%)=0.6551, A=2, B=32025.10 | 57.86 | — | — | — | — | — | — | — | — | — | — | — | — | 48.91 | 67.47 | 51.86 | 66.53 | |
| HydraLoRABackbone=LLaMA3-8B, Zero-shot=true, Params. (%)=0.5785, A=1, B=32025.10 | 57.78 | — | — | — | — | — | — | — | — | — | — | — | — | 49.7 | 66.79 | 51.84 | 66.08 | |
| BSLoRABackbone=LLaMA3-8B, Zero-shot=true, Params. (%)=0.2247, A=1, B=12025.10 | 57.74 | — | — | — | — | — | — | — | — | — | — | — | — | 48.72 | 67.24 | 52.14 | 65.98 | |
| LoRAr=8Backbone=LLaMA3-8B, Zero-shot=true, Params. (%)=0.2605, A=1, B=12025.10 | 57.61 | — | — | — | — | — | — | — | — | — | — | — | — | 49.03 | 66.59 | 52.01 | 65.92 | |
| CAT*Model Family=Zephyr-7B2025.10 | 57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CAT*Model Family=Mistral-7B2025.10 | 57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| WildJailbreakModel Family=Mistral-7B2025.10 | 57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Refusal TrainingModel Family=Zephyr-7B2025.10 | 56 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAP TrainedModel Family=Zephyr-7B2025.10 | 54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ASCoT (closed)Model Family=Zephyr-7B2025.10 | 54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ASCoT (open)Model Family=Zephyr-7B2025.10 | 54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| WildJailbreakModel Family=Zephyr-7B2025.10 | 53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NRGPTprotocol=five-shot generation, parameters=128M2025.12 | 29.3 | — | — | 30 | 30.5 | — | — | — | 35.5 | 32.1 | 31.5 | 29.2 | 19 | — | — | — | — | |
| GPTprotocol=five-shot generation, parameters=124M2025.12 | 28 | — | — | 26.7 | 31.2 | — | — | — | 36.8 | 29.5 | 26.2 | 27.3 | 18 | — | — | — | — | |
| RGPTprotocol=five-shot generation, parameters=126M2025.12 | 26 | — | — | 32.1 | 25.2 | — | — | — | 28.3 | 27.1 | 28 | 24.1 | 17 | — | — | — | — | |
| P-TuningBackbone=LLaMA3-8B, Zero-shot=true, Params. (%)=0.02802025.10 | 24.85 | — | — | — | — | — | — | — | — | — | — | — | — | 23.91 | 26.1 | 24.59 | 24.94 | |
| Prompt TuningBackbone=LLaMA3-8B, Zero-shot=true, Params. (%)=0.00042025.10 | 24.26 | — | — | — | — | — | — | — | — | — | — | — | — | 23.88 | 24.37 | 25.08 | 23.27 | |
| LLaMA3-8BBackbone=LLaMA3-8B, Zero-shot=true2025.10 | 23.12 | — | — | — | — | — | — | — | — | — | — | — | — | 21.82 | 21.71 | 24.29 | 24.04 | |
| AbstRaLModel=Qwen2.5-0.5B-Instruct, GSM Train Method=AbstRaL, Zero-shot=true2025.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | 38.8 | 53.5 | 42.2 | 50.7 | |
| AbstRaLModel=Qwen2.5-Math-7B-Instruct, GSM Train Method=AbstRaL, Zero-shot=true2025.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | 68.6 | 82.7 | 63.8 | 76.5 | |
| ChatGPTzero-shot=true2024.02 | — | 39 | 73.6 | 46.4 | 70.1 | 79 | — | — | — | — | — | — | — | — | — | — | — | |
| Claude2.1zero-shot=true2024.02 | — | 47 | 69.4 | 40.4 | 60.7 | 68 | — | — | — | — | — | — | — | — | — | — | — | |
| CoAModel=Qwen2.5-0.5B-Instruct, GSM Train Method=CoA, Zero-shot=true2025.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | 38.5 | 50.8 | 41.9 | 49.3 | |
| CoAModel=Qwen2.5-Math-7B-Instruct, GSM Train Method=CoA, Zero-shot=true2025.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | 68.5 | 81.7 | 63.3 | 75.5 | |
| CoT-RLModel=Qwen2.5-0.5B-Instruct, GSM Train Method=CoT-RL, Zero-shot=true2025.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | 38.5 | 50.8 | 41.2 | 49.1 | |
| CoT-RLModel=Qwen2.5-Math-7B-Instruct, GSM Train Method=CoT-RL, Zero-shot=true2025.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | 68.5 | 81.5 | 63 | 74.4 | |
| Geminizero-shot=true2024.02 | — | 35 | 82.6 | 54.4 | 69.3 | 78 | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4zero-shot=true2024.02 | — | 76 | 89.7 | 66.7 | 80.7 | 85 | — | — | — | — | — | — | — | — | — | — | — | |
| Llama2-13bzero-shot=true2024.02 | — | 31 | 70 | 30 | 43.6 | 65 | — | — | — | — | — | — | — | — | — | — | — | |
| Llama2-7bzero-shot=true2024.02 | — | 14 | 57.8 | 28.1 | 41.9 | 58 | — | — | — | — | — | — | — | — | — | — | — | |
| Llama3-8bzero-shot=true2024.02 | — | 34 | 76.9 | 50.9 | 65.9 | 74 | — | — | — | — | — | — | — | — | — | — | — | |
| Medical LLaMAModel Configuration=base model2026.02 | — | — | — | — | 39.3 | — | 49.6 | 50 | — | — | — | — | — | — | — | — | — | |
| Mistral-7bzero-shot=true2024.02 | — | 34 | 70.2 | 36.9 | 59.5 | 70 | — | — | — | — | — | — | — | — | — | — | — | |
| Mistral-8x7bzero-shot=true2024.02 | — | 41 | 74.4 | 53.5 | 67.6 | 74 | — | — | — | — | — | — | — | — | — | — | — | |
| Ori-SFTModel=Qwen2.5-0.5B-Instruct, GSM Train Method=Ori-SFT, Zero-shot=true2025.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | 37.5 | 51.5 | 41.6 | 49.7 | |
| Ori-SFTModel=Qwen2.5-Math-7B-Instruct, GSM Train Method=Ori-SFT, Zero-shot=true2025.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | 68.4 | 82.5 | 63.2 | 76.4 | |
| Transport and MergeModel Configuration=Fused w/o adaptation2026.02 | — | — | — | — | 39.7 | — | 50 | 49 | — | — | — | — | — | — | — | — | — | |
| Transport and MergeModel Configuration=Fused w/ adaptation2026.02 | — | — | — | — | 40.5 | — | 51.1 | 52 | — | — | — | — | — | — | — | — | — | |
| Yi-34Bzero-shot=true2024.02 | — | 33 | 81.8 | 52.6 | 67.6 | 77 | — | — | — | — | — | — | — | — | — | — | — | |
| Yi-6Bzero-shot=true2024.02 | — | 27 | 68.6 | 33.3 | 65.9 | 67 | — | — | — | — | — | — | — | — | — | — | — |