Code generation on MBPP (Acc, Cost)
90.8AccuracyNTele-R1-32B
Evaluation results
| Method | Links | ||
|---|---|---|---|
| NTele-R1-32B2025.08 | 90.8 | — | |
| DS-32B2025.08 | 89.6 | — | |
| COPESmall Model=EXAONE-3.5-2.4B, Large Model=GPT-4o, Shot=Zero, Train=x2025.06 | 66.4 | 1,279 | |
| COPESmall Model=Qwen-2.5-1.5B, Large Model=GPT-4o, Shot=Zero, Train=x2025.06 | 66.2 | 1,469 | |
| CascadeSmall Model=EXAONE-3.5-2.4B, Large Model=GPT-4o, Shot=Few, Train=x2025.06 | 64.6 | 2,706 | |
| LargeSmall Model=EXAONE-3.5-2.4B, Large Model=GPT-4o, Shot=Zero, Train=x2025.06 | 64 | 4,889 | |
| LargeSmall Model=Qwen-2.5-1.5B, Large Model=GPT-4o, Shot=Zero, Train=x2025.06 | 64 | 4,889 | |
| FrugalGPTSmall Model=EXAONE-3.5-2.4B, Large Model=GPT-4o, Shot=Few, Train=✓2025.06 | 63.6 | 4,012 | |
| ABCSmall Model=EXAONE-3.5-2.4B, Large Model=GPT-4o, Shot=Zero, Train=x2025.06 | 63.2 | 1,267 | |
| ABCSmall Model=Qwen-2.5-1.5B, Large Model=GPT-4o, Shot=Zero, Train=x2025.06 | 62.6 | 1,542 | |
| COPESmall Model=EXAONE-3.5-2.4B, Large Model=GPT-4o-mini, Shot=Zero, Train=x2025.06 | 62.2 | 77 | |
| CascadeSmall Model=Qwen-2.5-1.5B, Large Model=GPT-4o, Shot=Few, Train=x2025.06 | 62.2 | 2,263 | |
| COPESmall Model=Qwen-2.5-1.5B, Large Model=GPT-4o-mini, Shot=Zero, Train=x2025.06 | 61.4 | 94 | |
| ABCSmall Model=EXAONE-3.5-2.4B, Large Model=GPT-4o-mini, Shot=Zero, Train=x2025.06 | 59.4 | 80 | |
| FrugalGPTSmall Model=Qwen-2.5-1.5B, Large Model=GPT-4o, Shot=Few, Train=✓2025.06 | 59.4 | 5,029 | |
| CascadeSmall Model=EXAONE-3.5-2.4B, Large Model=GPT-4o-mini, Shot=Few, Train=x2025.06 | 59.2 | 159 | |
| ABCSmall Model=Qwen-2.5-1.5B, Large Model=GPT-4o-mini, Shot=Zero, Train=x2025.06 | 59.2 | 96 | |
| LargeSmall Model=EXAONE-3.5-2.4B, Large Model=GPT-4o-mini, Shot=Zero, Train=x2025.06 | 57.6 | 276 | |
| LargeSmall Model=Qwen-2.5-1.5B, Large Model=GPT-4o-mini, Shot=Zero, Train=x2025.06 | 57.6 | 276 | |
| CascadeSmall Model=Qwen-2.5-1.5B, Large Model=GPT-4o-mini, Shot=Few, Train=x2025.06 | 57.4 | 147 | |
| Small (multi)Small Model=EXAONE-3.5-2.4B, Shot=Zero, Train=x2025.06 | 57.2 | — | |
| FrugalGPTSmall Model=Qwen-2.5-1.5B, Large Model=GPT-4o-mini, Shot=Few, Train=✓2025.06 | 57.2 | 493 | |
| Small (multi)Small Model=Qwen-2.5-1.5B, Shot=Zero, Train=x2025.06 | 54.2 | — | |
| FrugalGPTSmall Model=EXAONE-3.5-2.4B, Large Model=GPT-4o-mini, Shot=Few, Train=✓2025.06 | 53.4 | 240 | |
| Small (single)Small Model=EXAONE-3.5-2.4B, Shot=Zero, Train=x2025.06 | 42.2 | — | |
| Small (single)Small Model=Qwen-2.5-1.5B, Shot=Zero, Train=x2025.06 | 38.6 | — |