Code Generation on LCB (Accuracy)
85.5AccuracyQwen-3.5 27B
Evaluation Results
| Method | Links | |
|---|---|---|
| Qwen-3.5 27BSpeedup @32k=0.55x, Speedup @16k=0.5x2026.04 | 85.5 | |
| MOPDModel=MiMo-V2-Flash2026.06 | 83.2 | |
| TeacherModel=MiMo-V2-Flash, Status=RL-trained2026.06 | 82.6 | |
| Apriel-1.6Speedup @32k=1.0x, Speedup @16k=1.0x2026.04 | 78.8 | |
| StudentModel=MiMo-V2-Flash2026.06 | 77.5 | |
| Nemotron-3-Nano 30BSpeedup @32k=4.09x, Speedup @16k=2.8x2026.04 | 75.3 | |
| Falcon-H1R 7BSpeedup @32k=4.61x, Speedup @16k=3.4x2026.04 | 74.3 | |
| Nemotron-Nano 12B v2Speedup @32k=5.85x, Speedup @16k=4.3x2026.04 | 70.9 | |
| Reg|Lklhd–26Speedup @32k=2.85x, Speedup @16k=1.5x2026.04 | 69.2 | |
| all-attentionSpeedup @32k=1.0x, Speedup @16k=1.0x2026.04 | 68.6 | |
| Idealized|All–18Speedup @32k=1.99x, Speedup @16k=1.1x2026.04 | 67.3 | |
| Reg|Lklhd–18Speedup @32k=4.76x, Speedup @16k=2.2x2026.04 | 64.8 | |
| Idealized|Lklhd–6Speedup @32k=6.2x, Speedup @16k=2.4x2026.04 | 57 | |
| Idealized|All–6Speedup @32k=6.13x, Speedup @16k=2.5x2026.04 | 55.9 | |
| Apriel-H1 15BSpeedup @32k=1.97x, Speedup @16k=1.9x2026.04 | 54 | |
| Reg|Lklhd–13Speedup @32k=6.9x, Speedup @16k=2.7x2026.04 | 52.6 | |
| Reg|Lklhd–10Speedup @32k=10.69x, Speedup @16k=4.2x2026.04 | 50.2 | |
| OLMo-Hybrid-Think 7BSpeedup @32k=2.51x, Speedup @16k=2.1x2026.04 | 48.6 | |
| S1: Distil. Idealized|All–6Speedup @32k=6.13x, Speedup @16k=2.5x2026.04 | 44.4 | |
| INTUITORModel=Llama3.2-3B-Ins2025.05 | 13.4 | |
| GRPOModel=Llama3.2-3B-Ins2025.05 | 12.7 | |
| GRPO-PVModel=Llama3.2-3B-Ins2025.05 | 10.9 | |
| BaselineModel=Llama3.2-3B-Ins2025.05 | 10.6 | |
| NITPModel scale=9bA1b, Evaluation=few-shot, Context length=81922026.05 | 8 | |
| NTPModel scale=9bA1b, Evaluation=few-shot, Context length=81922026.05 | 6.95 | |
| NITPModel scale=3bA0.5b, Evaluation=few-shot, Context length=81922026.05 | 4.17 | |
| NTPModel scale=3bA0.5b, Evaluation=few-shot, Context length=81922026.05 | 3.82 | |
| NITPModel scale=1.9bA0.3b, Evaluation=few-shot, Context length=81922026.05 | 2.26 | |
| NTPModel scale=1.9bA0.3b, Evaluation=few-shot, Context length=81922026.05 | 1.56 |