List your company
ResearchFormerly WizWand

General Knowledge on MMLU (test)

87.6AccuracySC-MAS
11.88831.54451.270.856Jul 20, 2025Sep 15, 2025Nov 12, 2025Jan 9, 2026Mar 8, 2026May 5, 2026Jul 2, 2026
Updated 4mo ago

Evaluation results

MethodLinks
SC-MASLLM backbone=LLM Pool, MAS=true, Routing=true2026.0187.6
MasRouterLLM backbone=LLM Pool, MAS=true, Routing=true2026.0184.25
GPTSwarmLLM backbone=gemini-1.5-flash, MAS=true, Routing=false2026.0183.22
AgentPruneLLM backbone=gemini-1.5-flash, MAS=true, Routing=false2026.0183.1
AFlowLLM backbone=gpt-4o-mini, MAS=true, Routing=false2026.0183.1
AgentPruneLLM backbone=gpt-4o-mini, MAS=true, Routing=false2026.0183.02
GPTSwarmLLM backbone=gpt-4o-mini, MAS=true, Routing=false2026.0182.8
AFlowLLM backbone=gemini-1.5-flash, MAS=true, Routing=false2026.0182.35
RouterDCLLM backbone=LLM Pool, MAS=false, Routing=true2026.0182.01
RouteLLMLLM backbone=LLM Pool, MAS=false, Routing=true2026.0181.04
VanillaLLM backbone=gemini-1.5-flash, MAS=false, Routing=false2026.0180.04
VanillaLLM backbone=llama-3.1-70b, MAS=false, Routing=false2026.0179.08
PromptLLMLLM backbone=LLM Pool, MAS=false, Routing=true2026.0178.43
VanillaLLM backbone=gpt-4o-mini, MAS=false, Routing=false2026.0177.81
TiDAR-8BNumber of shots=5, Evaluation protocol=Trust AR2026.0676.57
FrugalGPTLLM backbone=LLM Pool, MAS=false, Routing=true2026.0176.24
VDLMPost-train=SFT+RL, Model Type=Diffusion, Number of shots=5, Evaluation Protocol=same protocol (*)2026.0173.9
Qwen3-4B-BaseNumber of shots=52026.0671.76
Dream-7B-BaseNumber of shots=52026.0669.5
FReDA-4BNumber of shots=5, Evaluation protocol=Self-refine2026.0669.11
FReDA-4BNumber of shots=5, Evaluation protocol=Best-of-N2026.0669.11
LLaMA3 8BPost-train=SFT+RL, Model Type=Autoregressive, Number of shots=5, Evaluation Protocol=same protocol (*)2026.0168.4
VanillaLLM backbone=claude-3.5-haiku, MAS=false, Routing=false2026.0167.97
LLaDA-8B-BaseNumber of shots=52026.0665.9
Qwen2.5-3B-BaseNumber of shots=52026.0665.6
LLaDA 8BPost-train=SFT, Model Type=Diffusion, Number of shots=5, Evaluation Protocol=same protocol (*)2026.0165.5
LLaDA-MoE-7B-A1B-BaseNumber of shots=52026.0664.59
BlockDiff-4BNumber of shots=52026.0664.56
PARTREPAvg KV Cache=280.4, Avg Prefill FLOPs=2.481 T, Backbone=Qwen 2.5-3B, τ=0.152026.0763.2
No RepetitionAvg KV Cache=235.5, Avg Prefill FLOPs=1.549 T, Backbone=Qwen 2.5-3B2026.0762.5
LLMLinguaAvg KV Cache=277.4, Avg Prefill FLOPs=2.105 T, Backbone=Qwen 2.5-3B, Repetition Type=Appending summary2026.0761
Naïve SummaryAvg KV Cache=277.4, Avg Prefill FLOPs=3.676 T, Backbone=Qwen 2.5-3B, Repetition Type=Appending summary2026.0760.8
H2O EvictionAvg KV Cache=270.8, Avg Prefill FLOPs=3.162 T, Backbone=Qwen 2.5-3B, Repetition Type=Compressing full repetition2026.0760.5
Full RepetitionAvg KV Cache=471.1, Avg Prefill FLOPs=3.140 T, Backbone=Qwen 2.5-3B2026.0760.3
Echo EvictionAvg KV Cache=235.5, Avg Prefill FLOPs=3.152 T, Backbone=Qwen 2.5-3B, Repetition Type=Compressing full repetition2026.0760.3
Llama-3.3-70B-Baseshot=3-shot, approach=likelihood-based, access_level=Open-weights, region=Non-European2026.0254.9
Gemma-3-27B-Baseshot=3-shot, approach=likelihood-based, access_level=Open-weights, region=Non-European2026.0254.7
Qwen-3-14B-Baseshot=3-shot, approach=likelihood-based, access_level=Open-weights, region=Non-European2026.0254.2
Mistral-3.2-24B-Baseshot=3-shot, approach=likelihood-based, access_level=Open-weights, region=European2026.0253.9
Qwen-3-30B-A3B-Baseshot=3-shot, approach=likelihood-based, access_level=Open-weights, region=Non-European2026.0253
SASFTModel=Qwen3-8B-Base2025.0752.11
OLMo-3-32B-Baseshot=3-shot, approach=likelihood-based, access_level=Fully-open, region=Non-European2026.0252
Gemma-3-12B-Baseshot=3-shot, approach=likelihood-based, access_level=Open-weights, region=Non-European2026.0251.9
SFT+PenaltyModel=Qwen3-8B-Base2025.0751.56
SFTModel=Qwen3-8B-Base2025.0750.73
Apertus-70B-Baseshot=3-shot, approach=likelihood-based, access_level=Fully-open, region=European2026.0249.4
Deepseek 7BPost-train=SFT+RL, Model Type=Autoregressive, Number of shots=5, Evaluation Protocol=RL alignment (†)2026.0149.4
SFT+GRPOModel=Qwen3-8B-Base2025.0748.63
Apertus-8B-Baseshot=3-shot, approach=likelihood-based, access_level=Fully-open, region=European2026.0247.1
EuroLLM-22B-Baseshot=3-shot, approach=likelihood-based, access_level=Fully-open, region=European2026.0246.4
Llama-3.1-8B-Baseshot=3-shot, approach=likelihood-based, access_level=Open-weights, region=Non-European2026.0246.3
OLMo-3-7B-Baseshot=3-shot, approach=likelihood-based, access_level=Fully-open, region=Non-European2026.0246
SFTModel=Gemma-2-9B2025.0745.55
SFT+GRPOModel=Gemma-2-9B2025.0744.99
SFT+PenaltyModel=Gemma-2-9B2025.0744.51
EuroLLM-9B-Baseshot=3-shot, approach=likelihood-based, access_level=Fully-open, region=European2026.0243.9
SASFTModel=Gemma-2-9B2025.0743.55
SFT+PenaltyModel=Qwen3-1.7B-Base2025.0738.38
SASFTModel=Qwen3-1.7B-Base2025.0738.23
SFTModel=Qwen3-1.7B-Base2025.0738.06
SFT+GRPOModel=Qwen3-1.7B-Base2025.0737.88
SASFTModel=Llama-3.1-8B2025.0734.53
SFTModel=Llama-3.1-8B2025.0733.97
SFT+GRPOModel=Llama-3.1-8B2025.0733.71
SFT+PenaltyModel=Llama-3.1-8B2025.0733.29
SFTModel=Gemma-2-2B2025.0728.36
SFT+PenaltyModel=Gemma-2-2B2025.0728.32
SASFTModel=Gemma-2-2B2025.0728.09
SFT+GRPOModel=Gemma-2-2B2025.0728.04
DiffuLLaMA-7BNumber of shots=52026.0627.42
LLMLingua Comp.Avg KV Cache=78.6, Avg Prefill FLOPs=1.130 T, Backbone=Qwen 2.5-3B, Repetition Type=Compressing full repetition2026.0714.8