Code Generation on LiveCode (Pass@1)
42.8Pass@1Qwen 3.5
Evaluation results
| Method | Links | |
|---|---|---|
| Qwen 3.5Zero-shot=true, Parameters=122B-A10B, Organization=Alibaba Cloud2026.05 | 42.8 | |
| Nemotron 3 SuperZero-shot=true, Parameters=120B-A12B, Organization=Nvidia2026.05 | 41.26 | |
| GLM-4.5VZero-shot=true, Parameters=106B-A12B, Organization=Z.ai2026.05 | 34.15 | |
| Phoenix-VL 1.5 MediumZero-shot=true, Parameters=123B, Organization=HTX2026.05 | 32.4 | |
| Llama 4 MaverickZero-shot=true, Parameters=400B-A17B, Organization=MetaAI2026.05 | 26.46 | |
| GRIP 16BModel Size=16B, Total Budget=300B tokens2026.02 | 6.4 | |
| GRIP 8BModel Size=8B, Total Budget=300B tokens2026.02 | 5.3 | |
| 6. GRIP (Full)Model=GRIP-8B, Total Budget=100B tokens2026.02 | 2.9 | |
| 5. + DiversityModel=GRIP-8B, Total Budget=100B tokens2026.02 | 2.5 | |
| 4. + Loss ReplayModel=GRIP-8B, Total Budget=100B tokens2026.02 | 2.4 | |
| 3. + Static ReplayModel=GRIP-8B, Total Budget=100B tokens2026.02 | 1.8 | |
| Random 16BModel Size=16B, Total Budget=300B tokens2026.02 | 1.5 | |
| 2. + Static BudgetModel=GRIP-8B, Total Budget=100B tokens2026.02 | 1.3 | |
| Random 8BModel Size=8B, Total Budget=300B tokens2026.02 | 1.2 | |
| 1. RandomModel=GRIP-8B, Total Budget=100B tokens2026.02 | 0.8 |