Python Programming on MBPP
95.5MBPP ScoreOkshot
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Okshotk-shot=102026.06 | 95.5 | — | — | — | — | 0.286 | 67.3 | |
| SOTA LLM2026.06 | 86.2 | — | — | — | — | 0.001 | — | |
| Qwen3-4B + FBS-Full (ours)Backbone=Qwen3-4B2026.01 | 46.3 | 532 | 0.7 | 36 | — | — | — | |
| BaselineBase Model=Qwen1.5-MoE, Weight Bits=162026.04 | 38.2 | — | — | — | 46.1 | — | — | |
| GPTQBase Model=Qwen1.5-MoE, Weight Bits=32026.04 | 30.18 | — | — | — | 37.48 | — | — | |
| MoBiEBase Model=Qwen1.5-MoE, Weight Bits=1.352026.04 | 29.27 | — | — | — | 37.54 | — | — | |
| ARB-LLMBase Model=Qwen1.5-MoE, Weight Bits=1.112026.04 | 21.88 | — | — | — | 30.6 | — | — | |
| BaselineBase Model=OLMoE, Weight Bits=162026.04 | 21.8 | — | — | — | 33.45 | — | — | |
| BiLLMBase Model=Qwen1.5-MoE, Weight Bits=1.112026.04 | 19.41 | — | — | — | 28.8 | — | — | |
| NoWagBase Model=Qwen1.5-MoE, Weight Bits=2.042026.04 | 16.76 | — | — | — | 26.94 | — | — | |
| MoEQuantBase Model=Qwen1.5-MoE, Weight Bits=22026.04 | 14.32 | — | — | — | 22.56 | — | — | |
| MoBiEBase Model=OLMoE, Weight Bits=1.462026.04 | 12.12 | — | — | — | 24.54 | — | — | |
| GPTQBase Model=OLMoE, Weight Bits=32026.04 | 12 | — | — | — | 24.85 | — | — | |
| GPTQBase Model=Qwen1.5-MoE, Weight Bits=22026.04 | 9.24 | — | — | — | 16.96 | — | — | |
| ARB-LLMBase Model=OLMoE, Weight Bits=1.112026.04 | 6.32 | — | — | — | 16.97 | — | — | |
| AWQBase Model=Qwen1.5-MoE, Weight Bits=32026.04 | 6.31 | — | — | — | 15.1 | — | — | |
| BiLLMBase Model=OLMoE, Weight Bits=1.112026.04 | 3.41 | — | — | — | 14.85 | — | — | |
| NoWagBase Model=OLMoE, Weight Bits=2.042026.04 | 1.02 | — | — | — | 11.77 | — | — | |
| AWQBase Model=OLMoE, Weight Bits=32026.04 | 0 | — | — | — | 3.88 | — | — | |
| AWQBase Model=OLMoE, Weight Bits=22026.04 | 0 | — | — | — | 1.9 | — | — | |
| GPTQBase Model=OLMoE, Weight Bits=22026.04 | 0 | — | — | — | 6.07 | — | — | |
| MoEQuantBase Model=OLMoE, Weight Bits=22026.04 | 0 | — | — | — | 9.69 | — | — | |
| AWQBase Model=Qwen1.5-MoE, Weight Bits=22026.04 | 0 | — | — | — | 12.1 | — | — |