List your company
ResearchFormerly WizWand

Code Generation on LCB (Accuracy)

85.5AccuracyQwen-3.5 27B
-1.797620.866243.5366.1938May 26, 2025Jul 31, 2025Oct 6, 2025Dec 11, 2025Feb 16, 2026Apr 23, 2026Jun 29, 2026
Updated 4mo ago

Evaluation results

MethodLinks
Qwen-3.5 27BSpeedup @32k=0.55x, Speedup @16k=0.5x2026.0485.5
MOPDModel=MiMo-V2-Flash2026.0683.2
TeacherModel=MiMo-V2-Flash, Status=RL-trained2026.0682.6
Apriel-1.6Speedup @32k=1.0x, Speedup @16k=1.0x2026.0478.8
StudentModel=MiMo-V2-Flash2026.0677.5
Nemotron-3-Nano 30BSpeedup @32k=4.09x, Speedup @16k=2.8x2026.0475.3
Falcon-H1R 7BSpeedup @32k=4.61x, Speedup @16k=3.4x2026.0474.3
Nemotron-Nano 12B v2Speedup @32k=5.85x, Speedup @16k=4.3x2026.0470.9
Reg|Lklhd–26Speedup @32k=2.85x, Speedup @16k=1.5x2026.0469.2
all-attentionSpeedup @32k=1.0x, Speedup @16k=1.0x2026.0468.6
Idealized|All–18Speedup @32k=1.99x, Speedup @16k=1.1x2026.0467.3
Reg|Lklhd–18Speedup @32k=4.76x, Speedup @16k=2.2x2026.0464.8
Idealized|Lklhd–6Speedup @32k=6.2x, Speedup @16k=2.4x2026.0457
Idealized|All–6Speedup @32k=6.13x, Speedup @16k=2.5x2026.0455.9
Apriel-H1 15BSpeedup @32k=1.97x, Speedup @16k=1.9x2026.0454
Reg|Lklhd–13Speedup @32k=6.9x, Speedup @16k=2.7x2026.0452.6
Reg|Lklhd–10Speedup @32k=10.69x, Speedup @16k=4.2x2026.0450.2
OLMo-Hybrid-Think 7BSpeedup @32k=2.51x, Speedup @16k=2.1x2026.0448.6
S1: Distil. Idealized|All–6Speedup @32k=6.13x, Speedup @16k=2.5x2026.0444.4
INTUITORModel=Llama3.2-3B-Ins2025.0513.4
GRPOModel=Llama3.2-3B-Ins2025.0512.7
GRPO-PVModel=Llama3.2-3B-Ins2025.0510.9
BaselineModel=Llama3.2-3B-Ins2025.0510.6
NITPModel scale=9bA1b, Evaluation=few-shot, Context length=81922026.058
NTPModel scale=9bA1b, Evaluation=few-shot, Context length=81922026.056.95
NITPModel scale=3bA0.5b, Evaluation=few-shot, Context length=81922026.054.17
NTPModel scale=3bA0.5b, Evaluation=few-shot, Context length=81922026.053.82
NITPModel scale=1.9bA0.3b, Evaluation=few-shot, Context length=81922026.052.26
NTPModel scale=1.9bA0.3b, Evaluation=few-shot, Context length=81922026.051.56