Code Generation on MBPP+ (Accuracy & Improvement)
76.9AccuracyPrimitives-based MAS
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Primitives-based MASBackbone=Qwen3-14B2026.02 | 76.9 | 8.4 | |
| Primitives-based MASBackbone=Qwen3-4B2026.02 | 76.4 | 12.9 | |
| LatentMASBackbone=Qwen3-14B2026.02 | 75.7 | 7.2 | |
| VotingBackbone=Qwen3-14B2026.02 | 75.7 | 7.2 | |
| PlanningBackbone=Qwen3-14B2026.02 | 74.6 | 6.1 | |
| VotingBackbone=Qwen3-4B2026.02 | 73.8 | 10.3 | |
| ReviewBackbone=Qwen3-14B2026.02 | 73.8 | 5.3 | |
| LatentMASBackbone=Qwen3-4B2026.02 | 73.5 | 10 | |
| TextMASBackbone=Qwen3-14B2026.02 | 72.8 | 4.3 | |
| ReviewBackbone=Qwen3-4B2026.02 | 72.2 | 8.7 | |
| PlanningBackbone=Qwen3-4B2026.02 | 72.2 | 8.7 | |
| TextMASBackbone=Qwen3-4B2026.02 | 69.8 | 6.3 | |
| SingleBackbone=Qwen3-14B2026.02 | 68.5 | — | |
| Qwen3Parameters=8B2026.03 | 67.9 | — | |
| Qwen2.5-7BBase Model=Qwen2.5-7B2026.03 | 67.5 | — | |
| Qwen3-8BBase Model=Qwen3-8B2026.03 | 66.4 | — | |
| SDARBase Model=Qwen3-8B2026.03 | 66.4 | 0 | |
| TraDOBase Model=Qwen3-8B2026.03 | 66.4 | 0 | |
| Gemma3Parameters=12B2026.03 | 66.1 | — | |
| Qwen2.5Parameters=4B2026.03 | 65.9 | — | |
| LoopRPTParameters=2.6B2026.03 | 65.08 | — | |
| OuroParameters=2.6B2026.03 | 64.81 | — | |
| SingleBackbone=Qwen3-4B2026.02 | 63.5 | — | |
| Qwen2.5Parameters=7B2026.03 | 63.5 | — | |
| Gemma3Parameters=3B2026.03 | 54.2 | — | |
| dTRPOBase Model=Qwen2.5-7B2026.03 | 51.6 | 15.9 | |
| Llama3.1Parameters=8B2026.03 | 51.6 | — | |
| Qwen3Parameters=4B2026.03 | 51.1 | — | |
| Llama3.2Parameters=3B2026.03 | 39.7 | — |