List your company
ResearchFormerly WizWand

Code Generation on LiveCodeBench (Accuracy and Average Score)

88.6AccuracyIn-place
-3.54420.37844.368.222May 23, 2025Jul 23, 2025Sep 23, 2025Nov 23, 2025Jan 24, 2026Mar 26, 2026May 27, 2026
Updated 5mo ago

Evaluation results

MethodLinks
In-placeModel=Gemma-3-27B, Number of turns=42025.1088.6——
MemoryModel=Gemma-3-27B, Number of turns=42025.1086.8——
In-placeModel=Gemma-3-27B, Number of turns=32025.1086.8——
Multi-turnModel=Gemma-3-27B, Number of turns=42025.1085.6——
In-placeModel=Gemma-3-27B, Number of turns=22025.1085——
Multi-turnModel=Gemma-3-27B, Number of turns=32025.1084.4——
MemoryModel=Gemma-3-27B, Number of turns=32025.1084.4——
Multi-turnModel=Gemma-3-27B, Number of turns=22025.1080.8——
MemoryModel=Gemma-3-27B, Number of turns=22025.1080.8——
StepFlowBackbone=GPT-OSS-20B medium2026.0479.5——
In-placeModel=Llama-3.1-70B, Number of turns=42025.1077.3——
Multi-turnModel=Llama-3.1-70B, Number of turns=42025.1076.1——
Multi-turnModel=Gemma-3-27B, Number of turns=12025.1076.1——
MemoryModel=Gemma-3-27B, Number of turns=12025.1076.1——
In-placeModel=Gemma-3-27B, Number of turns=12025.1076.1——
Plan-and-Solve (PS+)Backbone=GPT-OSS-20B medium2026.0475.8——
In-placeModel=Llama-3.1-70B, Number of turns=32025.1074.3——
Multi-turnModel=Llama-3.1-70B, Number of turns=32025.1073.1——
In-placeModel=Llama-3.1-70B, Number of turns=22025.1071.9——
MemoryModel=Llama-3.1-70B, Number of turns=42025.1070.1——
GPT-OSS-20B mediumInference Protocol=Baseline2026.0470——
FullBase Model=DeepSeek-R1, Sparsity=50%2025.0569.1275.51—
Multi-turnModel=Llama-3.1-70B, Number of turns=22025.1068.9——
FullBase Model=openPangu-Ultra, Sparsity=31.25%2025.0567.6579.25—
PreMoEBase Model=openPangu-Ultra, Sparsity=31.25%2025.0566.9178.45-0.8
MemoryModel=Llama-3.1-70B, Number of turns=32025.1066.5——
PreMoEBase Model=DeepSeek-R1, Sparsity=50%2025.0566.3676.521.01
FullBase Model=Qwen3-30B-A3B, Sparsity=50%2025.0565.4480.69—
In-placeModel=Llama-3.1-70B, Number of turns=12025.1065.3——
PreMoEBase Model=Qwen3-30B-A3B, Sparsity=50%2025.0565.0779.92-0.77
MemoryModel=Llama-3.1-70B, Number of turns=22025.1063.5——
EASY-EPBase Model=openPangu-Ultra, Sparsity=31.25%2025.0562.3676.45-2.8
Multi-turnModel=Llama-3.1-70B, Number of turns=12025.1062.3——
Act-LogitsBase Model=openPangu-Ultra, Sparsity=31.25%2025.0561.7676.91-2.34
EASY-EPBase Model=DeepSeek-R1, Sparsity=50%2025.0561.1174.69-0.82
SEER (L)Base Model=openPangu-Ultra, Sparsity=31.25%2025.0558.8271.29-7.96
SEER (G)Base Model=openPangu-Ultra, Sparsity=31.25%2025.0558.8270.93-8.32
MemoryModel=Llama-3.1-70B, Number of turns=12025.1056.9——
EASY-EPBase Model=Qwen3-30B-A3B, Sparsity=50%2025.0556.4454.26-26.43
Multi-turnModel=Gemma-3-27B, Number of turns=02025.1054.5——
MemoryModel=Gemma-3-27B, Number of turns=02025.1054.5——
In-placeModel=Gemma-3-27B, Number of turns=02025.1054.5——
Act-LogitsBase Model=DeepSeek-R1, Sparsity=50%2025.0552.9462.87-12.64
SMCS2025.0752.17——
FrequencyBase Model=openPangu-Ultra, Sparsity=31.25%2025.0551.8469.67-9.58
Multi-turnModel=Llama-3.1-70B, Number of turns=02025.1049.1——
MemoryModel=Llama-3.1-70B, Number of turns=02025.1049.1——
In-placeModel=Llama-3.1-70B, Number of turns=02025.1049.1——
GPT-4.12025.0742.44——
MetaAgent-X RLTraining Paradigm=RL-based Auto MAS, Backbone=Qwen3 8B2026.0541——
QwQ-32B2025.0739.11——
MetaAgent-X SFTTraining Paradigm=RL-based Auto MAS, Backbone=Qwen3 8B2026.0536——
MetaAgent-X RLTraining Paradigm=RL-based Auto MAS2026.0536——
MetaAgent-X SFTTraining Paradigm=RL-based Auto MAS2026.0532——
Self-MoA2025.0729.35——
AFM-CoderTraining Paradigm=RL-based Auto MAS, Backbone=Qwen3 8B2026.0529.1——
AFlowTraining Paradigm=Search-based Auto MAS, Backbone=Qwen3 8B2026.0528.6——
AFlowTraining Paradigm=Search-based Auto MAS2026.0528——
RandomBase Model=DeepSeek-R1, Sparsity=50%2025.0527.4326.69-48.82
ScoreFlowTraining Paradigm=RL-based Auto MAS, Backbone=Qwen3 8B2026.0525.9——
RandomBase Model=openPangu-Ultra, Sparsity=31.25%2025.0525.7450.13-29.12
SA + GRPOTraining Paradigm=Single Agent, Backbone=Qwen3 8B2026.0525.7——
FrequencyBase Model=Qwen3-30B-A3B, Sparsity=50%2025.0525.3754.26-32.76
MaASTraining Paradigm=RL-based Auto MAS, Backbone=Qwen3 8B2026.0524.29——
MaASTraining Paradigm=RL-based Auto MAS2026.0524.29——
ScoreFlowTraining Paradigm=RL-based Auto MAS2026.0523.36——
SATraining Paradigm=Single Agent, Backbone=Qwen3 8B2026.0522.8——
ADASTraining Paradigm=Search-based Auto MAS, Backbone=Qwen3 8B2026.0520——
IBTPOBackbone=Llama3.1-8B-Instruct2026.0519.5——
Vanilla GRPOBackbone=Llama3.1-8B-Instruct2026.0517.8——
SA + GRPOTraining Paradigm=Single Agent2026.0516.7——
Initial ModelBackbone=Llama3.1-8B-Instruct2026.0516.1——
ADASTraining Paradigm=Search-based Auto MAS2026.0516——
SEER (G)Base Model=DeepSeek-R1, Sparsity=50%2025.0514.3424.31-51.2
SATraining Paradigm=Single Agent2026.0513.8——
SEER (L)Base Model=DeepSeek-R1, Sparsity=50%2025.058.0923.56-51.95
FrequencyBase Model=DeepSeek-R1, Sparsity=50%2025.055.8849.93-25.58
SEER (L)Base Model=Qwen3-30B-A3B, Sparsity=50%2025.052.210.64-80.05
SEER (G)Base Model=Qwen3-30B-A3B, Sparsity=50%2025.051.840.34-80.35
All-LogitsBase Model=openPangu-Ultra, Sparsity=31.25%2025.050.745.39-73.86
All-LogitsBase Model=DeepSeek-R1, Sparsity=50%2025.0505.46-70.05
RandomBase Model=Qwen3-30B-A3B, Sparsity=50%2025.05013.07-67.62
All-LogitsBase Model=Qwen3-30B-A3B, Sparsity=50%2025.0500.35-80.34
Act-LogitsBase Model=Qwen3-30B-A3B, Sparsity=50%2025.0500.82-79.87
AdamW2026.05—3.58—
Before RL (Base)Backbone=Qwen2.5-3B, Training Set=MATH2026.05—9.57—
Before RL (Base)Backbone=Qwen2.5-7B, Training Set=MATH2026.05—1.99—
GT-RewardBackbone=Qwen2.5-3B, Training Set=MATH2026.05—12.32—
GT-RewardBackbone=Qwen2.5-7B, Training Set=MATH2026.05—7.39—
INTUITORBackbone=Qwen2.5-3B, Training Set=MATH2026.05—14.88—
INTUITORBackbone=Qwen2.5-7B, Training Set=MATH2026.05—19.81—
MONA2026.05—5.02—
Muon2026.05—4.3—
VIGORBackbone=Qwen2.5-3B, Training Set=MATH2026.05—15.9—
VIGORBackbone=Qwen2.5-7B, Training Set=MATH2026.05—24.45—