Language Understanding on MMLU (test)
88MMLU Average AccuracyGPT 4o
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| GPT 4oModel Type=Proprietary2024.10 | 88 | — | — | — | — | — | — | — | |
| adv-ICLBackbone=ChatGPT, Shots=5-shot2023.12 | 74 | 61.3 | 76.7 | 82.3 | 75.8 | — | — | — | |
| OriginalModel=Qwen2.5-7B2026.04 | 74 | 70 | — | — | — | — | — | — | |
| FPFTModel=Qwen2.5-7B2026.04 | 74 | 70 | — | — | — | — | — | — | |
| H-CoTModel=Qwen2.5-7B2026.04 | 74 | 70 | — | — | — | — | — | — | |
| AutoRANModel=Qwen2.5-7B2026.04 | 74 | 70 | — | — | — | — | — | — | |
| Chain-of-LureModel=Qwen2.5-7B2026.04 | 74 | 70 | — | — | — | — | — | — | |
| FLAN-PALM 540Bfew-shot=true2022.05 | 73.5 | — | — | — | — | — | — | — | |
| LRFTModel=Qwen2.5-7B2026.04 | 73 | 70 | — | — | — | — | — | — | |
| CoT HijackingModel=Qwen2.5-7B2026.04 | 73 | 70 | — | — | — | — | — | — | |
| TSBH (Pass@1)Model=Qwen2.5-7B2026.04 | 73 | 69 | — | — | — | — | — | — | |
| TSBH (Pass@5)Model=Qwen2.5-7B2026.04 | 73 | 69 | — | — | — | — | — | — | |
| APOBackbone=ChatGPT, Shots=5-shot2023.12 | 72.1 | 58.3 | 75.6 | 80.7 | 73.9 | — | — | — | |
| Vanilla2025.05 | 72.06 | — | — | — | — | — | — | — | |
| CAA2025.05 | 71.88 | — | — | — | — | — | — | — | |
| SAEAXBENCH2025.05 | 71.77 | — | — | — | — | — | — | — | |
| STA2025.05 | 71.74 | — | — | — | — | — | — | — | |
| GPSBackbone=ChatGPT, Shots=5-shot2023.12 | 71.7 | 57.9 | 74.6 | 80 | 74.3 | — | — | — | |
| Prompt auto2025.05 | 71.59 | — | — | — | — | — | — | — | |
| Few-shotBackbone=ChatGPT, Shots=5-shot2023.12 | 71 | 57.5 | 73.9 | 79.2 | 73.5 | — | — | — | |
| VITAParameters=8x7B, Modality=Omni-modal2024.10 | 71 | — | — | — | — | — | — | — | |
| ROUGE-LBackbone=ChatGPT, Shots=5-shot2023.12 | 70.8 | 56.7 | 74.2 | 78.4 | 73.9 | — | — | — | |
| PerplexityBackbone=ChatGPT, Shots=5-shot2023.12 | 70.5 | 56.3 | 74.8 | 79.6 | 71.2 | — | — | — | |
| Qwen2-7BParameters=7B2024.07 | 70.3 | — | — | — | — | — | — | — | |
| Llama3-InstructParameters=8B, Modality=Pure text2024.10 | 67.1 | — | — | — | — | — | — | — | |
| Llama-3-8BParameters=8B2024.07 | 66.6 | — | — | — | — | — | — | — | |
| Baichuan-omniParameters=7B, Modality=Omni-modal2024.10 | 65.3 | — | — | — | — | — | — | — | |
| Gemma-7BParameters=7B2024.07 | 64.6 | — | — | — | — | — | — | — | |
| Mistral-7BParameters=7B2024.07 | 64.2 | — | — | — | — | — | — | — | |
| MistralSize=7B, prompting=few-shot chain-of-thought2024.02 | 62.4 | — | — | — | — | — | — | — | |
| GCAI2026.01 | 62.38 | 53.22 | 56 | 73.45 | 70.39 | — | — | — | |
| ICAI2026.01 | 62.05 | 53.28 | 55.07 | 73.38 | 70.29 | — | — | — | |
| Qwen1.5-ChatParameters=7B, Modality=Pure text2024.10 | 61.5 | — | — | — | — | — | — | — | |
| Qwen1.5-7BParameters=7B2024.07 | 61 | — | — | — | — | — | — | — | |
| Qwen3-1.7B-BaseMode=No Memorization (Baseline), Extra #Params=0, Relative Time=1x2026.02 | 60.4 | — | — | — | — | — | 0 | — | |
| Locas-GLUMode=TTT, Rank=512, Extra #Params=88.1M, Relative Time=3.7x2026.02 | 60.3 | — | — | — | — | — | -0.1 | — | |
| Locas-GLUMode=TTT, Extra #Params=11.0M, Relative Time=1.8x2026.02 | 60.2 | — | — | — | — | — | -0.2 | — | |
| OriginalModel=Llama3.2-3B2026.04 | 60 | 49 | — | — | — | — | — | — | |
| TempLoRAMode=TTT, Extra #Params=73.4M, Relative Time=4.7x2026.02 | 59.8 | — | — | — | — | — | -0.6 | — | |
| FLAN-PALM 62Bfew-shot=true2022.05 | 59.6 | — | — | — | — | — | — | — | |
| TempLoRAMode=TTT, Rank=512, Extra #Params=587M, Relative Time=11.2x2026.02 | 59.2 | — | — | — | — | — | -1.2 | — | |
| FPFTModel=Llama3.2-3B2026.04 | 59 | 49 | — | — | — | — | — | — | |
| LRFTModel=Llama3.2-3B2026.04 | 59 | 50 | — | — | — | — | — | — | |
| H-CoTModel=Llama3.2-3B2026.04 | 59 | 50 | — | — | — | — | — | — | |
| AutoRANModel=Llama3.2-3B2026.04 | 59 | 50 | — | — | — | — | — | — | |
| Chain-of-LureModel=Llama3.2-3B2026.04 | 59 | 49 | — | — | — | — | — | — | |
| CoT HijackingModel=Llama3.2-3B2026.04 | 59 | 49 | — | — | — | — | — | — | |
| TSBH (Pass@1)Model=Llama3.2-3B2026.04 | 59 | 50 | — | — | — | — | — | — | |
| TSBH (Pass@5)Model=Llama3.2-3B2026.04 | 59 | 50 | — | — | — | — | — | — | |
| LLM-QATPrecision (W-A-KV)=8-8-8, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 58.5 | 47.7 | 56.5 | 66.9 | 64.2 | — | — | — | |
| SmoothQuantPrecision (W-A-KV)=8-8-16, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 58.2 | 46.7 | 56 | 67.5 | 63.8 | — | — | — | |
| MAP-NeoParameters=7B, Modality=Pure text2024.10 | 58.2 | — | — | — | — | — | — | — | |
| FLAN-UL2 20Bfew-shot=true, checkpoin t=individual task best2022.05 | 58.1 | — | — | — | — | — | — | — | |
| adv-ICLBackbone=Vicuna v1.5, Shots=5-shot2023.12 | 58.1 | 44.1 | 58.9 | 64.8 | 64.5 | — | — | — | |
| SmoothQuantPrecision (W-A-KV)=8-8-8, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 58 | 46 | 56 | 67.3 | 64.1 | — | — | — | |
| LLaMA-30BPrecision (W-A-KV)=16-16-16, Size (GB)=60.6, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 57.8 | 46 | 55.8 | 66.7 | 63.4 | — | — | — | |
| RTNPrecision (W-A-KV)=8-8-16, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 57.8 | 45.6 | 56.3 | 66.8 | 63.7 | — | — | — | |
| RTNPrecision (W-A-KV)=8-8-8, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 57.5 | 45.8 | 55.6 | 66.3 | 63.4 | — | — | — | |
| LLM-QATPrecision (W-A-KV)=8-8-16, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 57.4 | 45.9 | 54.9 | 66.7 | 63.6 | — | — | — | |
| FLAN-UL2 20Bfew-shot=true, checkpoin t=best for both tasks2022.05 | 56.2 | — | — | — | — | — | — | — | |
| GPSBackbone=Vicuna v1.5, Shots=5-shot2023.12 | 55.9 | 40.4 | 56.9 | 64.1 | 62.3 | — | — | — | |
| APOBackbone=Vicuna v1.5, Shots=5-shot2023.12 | 55.9 | 40 | 57.2 | 63.7 | 62.7 | — | — | — | |
| PerplexityBackbone=Vicuna v1.5, Shots=5-shot2023.12 | 55.2 | 39.5 | 55.2 | 64.1 | 61.9 | — | — | — | |
| LLM-QATPrecision (W-A-KV)=4-8-16, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 55.1 | 44.4 | 52.8 | 63.6 | 61.2 | — | — | — | |
| ROUGE-LBackbone=Vicuna v1.5, Shots=5-shot2023.12 | 55 | 39.5 | 55.5 | 63.7 | 61.1 | — | — | — | |
| FP16Model=LLaMA-2-13B, Quantization=None, Calibration=FP162026.04 | 55 | — | — | — | — | — | — | — | |
| DeepSeekMath-BaseSize=7B, prompting=few-shot chain-of-thought2024.02 | 54.9 | — | — | — | — | — | — | — | |
| Few-shotBackbone=Vicuna v1.5, Shots=5-shot2023.12 | 54.6 | 38.7 | 55.8 | 63.3 | 61.5 | — | — | — | |
| COVERCALModel=LLaMA-2-13B, Quantization=AWQ INT4, Calibration Samples=128, Calibration=COVERCAL2026.04 | 54.6 | — | — | — | — | — | — | — | |
| LLM-QATPrecision (W-A-KV)=4-8-8, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 54.4 | 44.3 | 52.2 | 61.4 | 61 | — | — | — | |
| RTNPrecision (W-A-KV)=4-8-16, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 54.2 | 44 | 50.9 | 62.8 | 61.3 | — | — | — | |
| Max-ActVarModel=LLaMA-2-13B, Quantization=AWQ INT4, Calibration Samples=128, Calibration=Max-ActVar2026.04 | 54 | — | — | — | — | — | — | — | |
| RTNPrecision (W-A-KV)=4-8-8, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 53.9 | 43.6 | 51 | 62.2 | 60.6 | — | — | — | |
| FLAN T5 XXLfew-shot=true2022.05 | 53.7 | — | — | — | — | — | — | — | |
| RandomModel=LLaMA-2-13B, Quantization=AWQ INT4, Calibration Samples=128, Calibration=Random2026.04 | 53.5 | — | — | — | — | — | — | — | |
| LLM-QATPrecision (W-A-KV)=8-8-4, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 52.7 | 42.2 | 49.7 | 60.8 | 59.7 | — | — | — | |
| LLM-QATPrecision (W-A-KV)=4-6-16, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 51 | 41.7 | 47.7 | 58.9 | 57.5 | — | — | — | |
| OriginalModel=DeepSeek-7B2026.04 | 51 | 40 | — | — | — | — | — | — | |
| FPFTModel=DeepSeek-7B2026.04 | 51 | 40 | — | — | — | — | — | — | |
| H-CoTModel=DeepSeek-7B2026.04 | 51 | 39 | — | — | — | — | — | — | |
| AutoRANModel=DeepSeek-7B2026.04 | 51 | 39 | — | — | — | — | — | — | |
| Chain-of-LureModel=DeepSeek-7B2026.04 | 51 | 40 | — | — | — | — | — | — | |
| CoT HijackingModel=DeepSeek-7B2026.04 | 51 | 40 | — | — | — | — | — | — | |
| LRFTModel=DeepSeek-7B2026.04 | 50 | 39 | — | — | — | — | — | — | |
| TSBH (Pass@1)Model=DeepSeek-7B2026.04 | 50 | 40 | — | — | — | — | — | — | |
| TSBH (Pass@5)Model=DeepSeek-7B2026.04 | 50 | 40 | — | — | — | — | — | — | |
| LLM-QATPrecision (W-A-KV)=4-8-4, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 49.3 | 40.4 | 47.6 | 55.9 | 54.5 | — | — | — | |
| DeepSeek-Coder-Base-v1.5Size=7B, prompting=few-shot chain-of-thought2024.02 | 49.1 | — | — | — | — | — | — | — | |
| AdaGCSteps (tokens)=60k (1T), AdamW eps=1e-152025.02 | 48.7 | — | — | — | — | — | — | — | |
| GlobalGCSteps (tokens)=60k (1T), AdamW eps=1e-82025.02 | 48.61 | — | — | — | — | — | — | — | |
| GlobalGCSteps (tokens)=60k (1T), AdamW eps=1e-152025.02 | 48.48 | — | — | — | — | — | — | — | |
| METRO-T0++LARGE++Params=775M2023.05 | 48 | — | — | — | — | — | — | — | |
| GlobalGCSteps (tokens)=21k (350B), AdamW eps=1e-82025.02 | 47.75 | — | — | — | — | — | — | — | |
| OPT IML-Max 175Bfew-shot=true2022.05 | 47.1 | — | — | — | — | — | — | — | |
| LLaMA-13BPrecision (W-A-KV)=16-16-16, Size (GB)=24.2, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 46.7 | 36.2 | 44.4 | 54.3 | 53.3 | — | — | — | |
| SmoothQuantPrecision (W-A-KV)=8-8-16, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 46.7 | 36.4 | 44.5 | 53.7 | 53.4 | — | — | — | |
| SmoothQuantPrecision (W-A-KV)=8-8-8, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 46.6 | 36.1 | 44.5 | 53.5 | 53.3 | — | — | — | |
| LLM-QATPrecision (W-A-KV)=8-8-16, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 46.3 | 36.1 | 43.6 | 53.8 | 53.2 | — | — | — | |
| RTNPrecision (W-A-KV)=8-8-8, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 45.9 | 35.6 | 44.2 | 52.2 | 52.5 | — | — | — | |
| LLM-QATPrecision (W-A-KV)=8-8-8, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 45.8 | 36.1 | 43.5 | 52.6 | 52.5 | — | — | — |