Code Generation on LiveCodeBench (Accuracy and Average Score)
88.6AccuracyIn-place
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| In-placeModel=Gemma-3-27B, Number of turns=42025.10 | 88.6 | — | — | |
| MemoryModel=Gemma-3-27B, Number of turns=42025.10 | 86.8 | — | — | |
| In-placeModel=Gemma-3-27B, Number of turns=32025.10 | 86.8 | — | — | |
| Multi-turnModel=Gemma-3-27B, Number of turns=42025.10 | 85.6 | — | — | |
| In-placeModel=Gemma-3-27B, Number of turns=22025.10 | 85 | — | — | |
| Multi-turnModel=Gemma-3-27B, Number of turns=32025.10 | 84.4 | — | — | |
| MemoryModel=Gemma-3-27B, Number of turns=32025.10 | 84.4 | — | — | |
| Multi-turnModel=Gemma-3-27B, Number of turns=22025.10 | 80.8 | — | — | |
| MemoryModel=Gemma-3-27B, Number of turns=22025.10 | 80.8 | — | — | |
| StepFlowBackbone=GPT-OSS-20B medium2026.04 | 79.5 | — | — | |
| In-placeModel=Llama-3.1-70B, Number of turns=42025.10 | 77.3 | — | — | |
| Multi-turnModel=Llama-3.1-70B, Number of turns=42025.10 | 76.1 | — | — | |
| Multi-turnModel=Gemma-3-27B, Number of turns=12025.10 | 76.1 | — | — | |
| MemoryModel=Gemma-3-27B, Number of turns=12025.10 | 76.1 | — | — | |
| In-placeModel=Gemma-3-27B, Number of turns=12025.10 | 76.1 | — | — | |
| Plan-and-Solve (PS+)Backbone=GPT-OSS-20B medium2026.04 | 75.8 | — | — | |
| In-placeModel=Llama-3.1-70B, Number of turns=32025.10 | 74.3 | — | — | |
| Multi-turnModel=Llama-3.1-70B, Number of turns=32025.10 | 73.1 | — | — | |
| In-placeModel=Llama-3.1-70B, Number of turns=22025.10 | 71.9 | — | — | |
| MemoryModel=Llama-3.1-70B, Number of turns=42025.10 | 70.1 | — | — | |
| GPT-OSS-20B mediumInference Protocol=Baseline2026.04 | 70 | — | — | |
| FullBase Model=DeepSeek-R1, Sparsity=50%2025.05 | 69.12 | 75.51 | — | |
| Multi-turnModel=Llama-3.1-70B, Number of turns=22025.10 | 68.9 | — | — | |
| FullBase Model=openPangu-Ultra, Sparsity=31.25%2025.05 | 67.65 | 79.25 | — | |
| PreMoEBase Model=openPangu-Ultra, Sparsity=31.25%2025.05 | 66.91 | 78.45 | -0.8 | |
| MemoryModel=Llama-3.1-70B, Number of turns=32025.10 | 66.5 | — | — | |
| PreMoEBase Model=DeepSeek-R1, Sparsity=50%2025.05 | 66.36 | 76.52 | 1.01 | |
| FullBase Model=Qwen3-30B-A3B, Sparsity=50%2025.05 | 65.44 | 80.69 | — | |
| In-placeModel=Llama-3.1-70B, Number of turns=12025.10 | 65.3 | — | — | |
| PreMoEBase Model=Qwen3-30B-A3B, Sparsity=50%2025.05 | 65.07 | 79.92 | -0.77 | |
| MemoryModel=Llama-3.1-70B, Number of turns=22025.10 | 63.5 | — | — | |
| EASY-EPBase Model=openPangu-Ultra, Sparsity=31.25%2025.05 | 62.36 | 76.45 | -2.8 | |
| Multi-turnModel=Llama-3.1-70B, Number of turns=12025.10 | 62.3 | — | — | |
| Act-LogitsBase Model=openPangu-Ultra, Sparsity=31.25%2025.05 | 61.76 | 76.91 | -2.34 | |
| EASY-EPBase Model=DeepSeek-R1, Sparsity=50%2025.05 | 61.11 | 74.69 | -0.82 | |
| SEER (L)Base Model=openPangu-Ultra, Sparsity=31.25%2025.05 | 58.82 | 71.29 | -7.96 | |
| SEER (G)Base Model=openPangu-Ultra, Sparsity=31.25%2025.05 | 58.82 | 70.93 | -8.32 | |
| MemoryModel=Llama-3.1-70B, Number of turns=12025.10 | 56.9 | — | — | |
| EASY-EPBase Model=Qwen3-30B-A3B, Sparsity=50%2025.05 | 56.44 | 54.26 | -26.43 | |
| Multi-turnModel=Gemma-3-27B, Number of turns=02025.10 | 54.5 | — | — | |
| MemoryModel=Gemma-3-27B, Number of turns=02025.10 | 54.5 | — | — | |
| In-placeModel=Gemma-3-27B, Number of turns=02025.10 | 54.5 | — | — | |
| Act-LogitsBase Model=DeepSeek-R1, Sparsity=50%2025.05 | 52.94 | 62.87 | -12.64 | |
| SMCS2025.07 | 52.17 | — | — | |
| FrequencyBase Model=openPangu-Ultra, Sparsity=31.25%2025.05 | 51.84 | 69.67 | -9.58 | |
| Multi-turnModel=Llama-3.1-70B, Number of turns=02025.10 | 49.1 | — | — | |
| MemoryModel=Llama-3.1-70B, Number of turns=02025.10 | 49.1 | — | — | |
| In-placeModel=Llama-3.1-70B, Number of turns=02025.10 | 49.1 | — | — | |
| GPT-4.12025.07 | 42.44 | — | — | |
| MetaAgent-X RLTraining Paradigm=RL-based Auto MAS, Backbone=Qwen3 8B2026.05 | 41 | — | — | |
| QwQ-32B2025.07 | 39.11 | — | — | |
| MetaAgent-X SFTTraining Paradigm=RL-based Auto MAS, Backbone=Qwen3 8B2026.05 | 36 | — | — | |
| MetaAgent-X RLTraining Paradigm=RL-based Auto MAS2026.05 | 36 | — | — | |
| MetaAgent-X SFTTraining Paradigm=RL-based Auto MAS2026.05 | 32 | — | — | |
| Self-MoA2025.07 | 29.35 | — | — | |
| AFM-CoderTraining Paradigm=RL-based Auto MAS, Backbone=Qwen3 8B2026.05 | 29.1 | — | — | |
| AFlowTraining Paradigm=Search-based Auto MAS, Backbone=Qwen3 8B2026.05 | 28.6 | — | — | |
| AFlowTraining Paradigm=Search-based Auto MAS2026.05 | 28 | — | — | |
| RandomBase Model=DeepSeek-R1, Sparsity=50%2025.05 | 27.43 | 26.69 | -48.82 | |
| ScoreFlowTraining Paradigm=RL-based Auto MAS, Backbone=Qwen3 8B2026.05 | 25.9 | — | — | |
| RandomBase Model=openPangu-Ultra, Sparsity=31.25%2025.05 | 25.74 | 50.13 | -29.12 | |
| SA + GRPOTraining Paradigm=Single Agent, Backbone=Qwen3 8B2026.05 | 25.7 | — | — | |
| FrequencyBase Model=Qwen3-30B-A3B, Sparsity=50%2025.05 | 25.37 | 54.26 | -32.76 | |
| MaASTraining Paradigm=RL-based Auto MAS, Backbone=Qwen3 8B2026.05 | 24.29 | — | — | |
| MaASTraining Paradigm=RL-based Auto MAS2026.05 | 24.29 | — | — | |
| ScoreFlowTraining Paradigm=RL-based Auto MAS2026.05 | 23.36 | — | — | |
| SATraining Paradigm=Single Agent, Backbone=Qwen3 8B2026.05 | 22.8 | — | — | |
| ADASTraining Paradigm=Search-based Auto MAS, Backbone=Qwen3 8B2026.05 | 20 | — | — | |
| IBTPOBackbone=Llama3.1-8B-Instruct2026.05 | 19.5 | — | — | |
| Vanilla GRPOBackbone=Llama3.1-8B-Instruct2026.05 | 17.8 | — | — | |
| SA + GRPOTraining Paradigm=Single Agent2026.05 | 16.7 | — | — | |
| Initial ModelBackbone=Llama3.1-8B-Instruct2026.05 | 16.1 | — | — | |
| ADASTraining Paradigm=Search-based Auto MAS2026.05 | 16 | — | — | |
| SEER (G)Base Model=DeepSeek-R1, Sparsity=50%2025.05 | 14.34 | 24.31 | -51.2 | |
| SATraining Paradigm=Single Agent2026.05 | 13.8 | — | — | |
| SEER (L)Base Model=DeepSeek-R1, Sparsity=50%2025.05 | 8.09 | 23.56 | -51.95 | |
| FrequencyBase Model=DeepSeek-R1, Sparsity=50%2025.05 | 5.88 | 49.93 | -25.58 | |
| SEER (L)Base Model=Qwen3-30B-A3B, Sparsity=50%2025.05 | 2.21 | 0.64 | -80.05 | |
| SEER (G)Base Model=Qwen3-30B-A3B, Sparsity=50%2025.05 | 1.84 | 0.34 | -80.35 | |
| All-LogitsBase Model=openPangu-Ultra, Sparsity=31.25%2025.05 | 0.74 | 5.39 | -73.86 | |
| All-LogitsBase Model=DeepSeek-R1, Sparsity=50%2025.05 | 0 | 5.46 | -70.05 | |
| RandomBase Model=Qwen3-30B-A3B, Sparsity=50%2025.05 | 0 | 13.07 | -67.62 | |
| All-LogitsBase Model=Qwen3-30B-A3B, Sparsity=50%2025.05 | 0 | 0.35 | -80.34 | |
| Act-LogitsBase Model=Qwen3-30B-A3B, Sparsity=50%2025.05 | 0 | 0.82 | -79.87 | |
| AdamW2026.05 | — | 3.58 | — | |
| Before RL (Base)Backbone=Qwen2.5-3B, Training Set=MATH2026.05 | — | 9.57 | — | |
| Before RL (Base)Backbone=Qwen2.5-7B, Training Set=MATH2026.05 | — | 1.99 | — | |
| GT-RewardBackbone=Qwen2.5-3B, Training Set=MATH2026.05 | — | 12.32 | — | |
| GT-RewardBackbone=Qwen2.5-7B, Training Set=MATH2026.05 | — | 7.39 | — | |
| INTUITORBackbone=Qwen2.5-3B, Training Set=MATH2026.05 | — | 14.88 | — | |
| INTUITORBackbone=Qwen2.5-7B, Training Set=MATH2026.05 | — | 19.81 | — | |
| MONA2026.05 | — | 5.02 | — | |
| Muon2026.05 | — | 4.3 | — | |
| VIGORBackbone=Qwen2.5-3B, Training Set=MATH2026.05 | — | 15.9 | — | |
| VIGORBackbone=Qwen2.5-7B, Training Set=MATH2026.05 | — | 24.45 | — |