General Knowledge on MMLU (test)
87.6AccuracySC-MAS
Evaluation Results
| Method | Links | |
|---|---|---|
| SC-MASLLM backbone=LLM Pool, MAS=true, Routing=true2026.01 | 87.6 | |
| MasRouterLLM backbone=LLM Pool, MAS=true, Routing=true2026.01 | 84.25 | |
| GPTSwarmLLM backbone=gemini-1.5-flash, MAS=true, Routing=false2026.01 | 83.22 | |
| AgentPruneLLM backbone=gemini-1.5-flash, MAS=true, Routing=false2026.01 | 83.1 | |
| AFlowLLM backbone=gpt-4o-mini, MAS=true, Routing=false2026.01 | 83.1 | |
| AgentPruneLLM backbone=gpt-4o-mini, MAS=true, Routing=false2026.01 | 83.02 | |
| GPTSwarmLLM backbone=gpt-4o-mini, MAS=true, Routing=false2026.01 | 82.8 | |
| AFlowLLM backbone=gemini-1.5-flash, MAS=true, Routing=false2026.01 | 82.35 | |
| RouterDCLLM backbone=LLM Pool, MAS=false, Routing=true2026.01 | 82.01 | |
| RouteLLMLLM backbone=LLM Pool, MAS=false, Routing=true2026.01 | 81.04 | |
| VanillaLLM backbone=gemini-1.5-flash, MAS=false, Routing=false2026.01 | 80.04 | |
| VanillaLLM backbone=llama-3.1-70b, MAS=false, Routing=false2026.01 | 79.08 | |
| PromptLLMLLM backbone=LLM Pool, MAS=false, Routing=true2026.01 | 78.43 | |
| VanillaLLM backbone=gpt-4o-mini, MAS=false, Routing=false2026.01 | 77.81 | |
| TiDAR-8BNumber of shots=5, Evaluation protocol=Trust AR2026.06 | 76.57 | |
| FrugalGPTLLM backbone=LLM Pool, MAS=false, Routing=true2026.01 | 76.24 | |
| VDLMPost-train=SFT+RL, Model Type=Diffusion, Number of shots=5, Evaluation Protocol=same protocol (*)2026.01 | 73.9 | |
| Qwen3-4B-BaseNumber of shots=52026.06 | 71.76 | |
| Dream-7B-BaseNumber of shots=52026.06 | 69.5 | |
| FReDA-4BNumber of shots=5, Evaluation protocol=Self-refine2026.06 | 69.11 | |
| FReDA-4BNumber of shots=5, Evaluation protocol=Best-of-N2026.06 | 69.11 | |
| LLaMA3 8BPost-train=SFT+RL, Model Type=Autoregressive, Number of shots=5, Evaluation Protocol=same protocol (*)2026.01 | 68.4 | |
| VanillaLLM backbone=claude-3.5-haiku, MAS=false, Routing=false2026.01 | 67.97 | |
| LLaDA-8B-BaseNumber of shots=52026.06 | 65.9 | |
| Qwen2.5-3B-BaseNumber of shots=52026.06 | 65.6 | |
| LLaDA 8BPost-train=SFT, Model Type=Diffusion, Number of shots=5, Evaluation Protocol=same protocol (*)2026.01 | 65.5 | |
| LLaDA-MoE-7B-A1B-BaseNumber of shots=52026.06 | 64.59 | |
| BlockDiff-4BNumber of shots=52026.06 | 64.56 | |
| PARTREPAvg KV Cache=280.4, Avg Prefill FLOPs=2.481 T, Backbone=Qwen 2.5-3B, τ=0.152026.07 | 63.2 | |
| No RepetitionAvg KV Cache=235.5, Avg Prefill FLOPs=1.549 T, Backbone=Qwen 2.5-3B2026.07 | 62.5 | |
| LLMLinguaAvg KV Cache=277.4, Avg Prefill FLOPs=2.105 T, Backbone=Qwen 2.5-3B, Repetition Type=Appending summary2026.07 | 61 | |
| Naïve SummaryAvg KV Cache=277.4, Avg Prefill FLOPs=3.676 T, Backbone=Qwen 2.5-3B, Repetition Type=Appending summary2026.07 | 60.8 | |
| H2O EvictionAvg KV Cache=270.8, Avg Prefill FLOPs=3.162 T, Backbone=Qwen 2.5-3B, Repetition Type=Compressing full repetition2026.07 | 60.5 | |
| Full RepetitionAvg KV Cache=471.1, Avg Prefill FLOPs=3.140 T, Backbone=Qwen 2.5-3B2026.07 | 60.3 | |
| Echo EvictionAvg KV Cache=235.5, Avg Prefill FLOPs=3.152 T, Backbone=Qwen 2.5-3B, Repetition Type=Compressing full repetition2026.07 | 60.3 | |
| Llama-3.3-70B-Baseshot=3-shot, approach=likelihood-based, access_level=Open-weights, region=Non-European2026.02 | 54.9 | |
| Gemma-3-27B-Baseshot=3-shot, approach=likelihood-based, access_level=Open-weights, region=Non-European2026.02 | 54.7 | |
| Qwen-3-14B-Baseshot=3-shot, approach=likelihood-based, access_level=Open-weights, region=Non-European2026.02 | 54.2 | |
| Mistral-3.2-24B-Baseshot=3-shot, approach=likelihood-based, access_level=Open-weights, region=European2026.02 | 53.9 | |
| Qwen-3-30B-A3B-Baseshot=3-shot, approach=likelihood-based, access_level=Open-weights, region=Non-European2026.02 | 53 | |
| SASFTModel=Qwen3-8B-Base2025.07 | 52.11 | |
| OLMo-3-32B-Baseshot=3-shot, approach=likelihood-based, access_level=Fully-open, region=Non-European2026.02 | 52 | |
| Gemma-3-12B-Baseshot=3-shot, approach=likelihood-based, access_level=Open-weights, region=Non-European2026.02 | 51.9 | |
| SFT+PenaltyModel=Qwen3-8B-Base2025.07 | 51.56 | |
| SFTModel=Qwen3-8B-Base2025.07 | 50.73 | |
| Apertus-70B-Baseshot=3-shot, approach=likelihood-based, access_level=Fully-open, region=European2026.02 | 49.4 | |
| Deepseek 7BPost-train=SFT+RL, Model Type=Autoregressive, Number of shots=5, Evaluation Protocol=RL alignment (†)2026.01 | 49.4 | |
| SFT+GRPOModel=Qwen3-8B-Base2025.07 | 48.63 | |
| Apertus-8B-Baseshot=3-shot, approach=likelihood-based, access_level=Fully-open, region=European2026.02 | 47.1 | |
| EuroLLM-22B-Baseshot=3-shot, approach=likelihood-based, access_level=Fully-open, region=European2026.02 | 46.4 | |
| Llama-3.1-8B-Baseshot=3-shot, approach=likelihood-based, access_level=Open-weights, region=Non-European2026.02 | 46.3 | |
| OLMo-3-7B-Baseshot=3-shot, approach=likelihood-based, access_level=Fully-open, region=Non-European2026.02 | 46 | |
| SFTModel=Gemma-2-9B2025.07 | 45.55 | |
| SFT+GRPOModel=Gemma-2-9B2025.07 | 44.99 | |
| SFT+PenaltyModel=Gemma-2-9B2025.07 | 44.51 | |
| EuroLLM-9B-Baseshot=3-shot, approach=likelihood-based, access_level=Fully-open, region=European2026.02 | 43.9 | |
| SASFTModel=Gemma-2-9B2025.07 | 43.55 | |
| SFT+PenaltyModel=Qwen3-1.7B-Base2025.07 | 38.38 | |
| SASFTModel=Qwen3-1.7B-Base2025.07 | 38.23 | |
| SFTModel=Qwen3-1.7B-Base2025.07 | 38.06 | |
| SFT+GRPOModel=Qwen3-1.7B-Base2025.07 | 37.88 | |
| SASFTModel=Llama-3.1-8B2025.07 | 34.53 | |
| SFTModel=Llama-3.1-8B2025.07 | 33.97 | |
| SFT+GRPOModel=Llama-3.1-8B2025.07 | 33.71 | |
| SFT+PenaltyModel=Llama-3.1-8B2025.07 | 33.29 | |
| SFTModel=Gemma-2-2B2025.07 | 28.36 | |
| SFT+PenaltyModel=Gemma-2-2B2025.07 | 28.32 | |
| SASFTModel=Gemma-2-2B2025.07 | 28.09 | |
| SFT+GRPOModel=Gemma-2-2B2025.07 | 28.04 | |
| DiffuLLaMA-7BNumber of shots=52026.06 | 27.42 | |
| LLMLingua Comp.Avg KV Cache=78.6, Avg Prefill FLOPs=1.130 T, Backbone=Qwen 2.5-3B, Repetition Type=Compressing full repetition2026.07 | 14.8 |