Code Generation on LiveCodeBench (Difficulty Breakdown)
87.24Overall AccuracyHEART
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| HEARTModel=Gemini 3 Pro, Strategy=HEART2025.09 | 87.24 | — | — | — | |
| CoTModel=Gemini 3 Pro, Strategy=CoT2025.09 | 86.35 | — | — | — | |
| WaitModel=Gemini 3 Pro, Strategy=Wait2025.09 | 86.31 | — | — | — | |
| Self ReflectionModel=Gemini 3 Pro, Strategy=Self Reflection2025.09 | 86.01 | — | — | — | |
| Self ReflectionModel=Gemini 3 Flash, Strategy=Self Reflection2025.09 | 85.76 | — | — | — | |
| DAJInference Protocol=Test-time Scaling2026.01 | 84.7 | 100 | 89.7 | 73.8 | |
| CoTModel=Gemini 3 Flash, Strategy=CoT2025.09 | 84.57 | — | — | — | |
| WaitModel=Gemini 3 Flash, Strategy=Wait2025.09 | 83.38 | — | — | — | |
| HEARTModel=Gemini 3 Flash, Strategy=HEART2025.09 | 83.38 | — | — | — | |
| DeepSeek GRMInference Protocol=Test-time Scaling2026.01 | 83.2 | 100 | 94.9 | 67.2 | |
| o4-mini (high)Inference Protocol=Zero-shot2026.01 | 77.1 | 100 | 89.7 | 57.4 | |
| RandomInference Protocol=Test-time Scaling2026.01 | 76 | 100 | 89.7 | 54.9 | |
| Gemini-2.5-Pro-06-05Inference Protocol=Zero-shot2026.01 | 72.5 | 100 | 82.1 | 52.5 | |
| o4-mini (medium)Inference Protocol=Zero-shot2026.01 | 71.8 | 96.8 | 79.5 | 54.1 | |
| Skywork-o1 PRMInference Protocol=Test-time Scaling2026.01 | 68.7 | 100 | 84.6 | 42.6 | |
| AZR-R-TAPBase Model Type=Coder2026.03 | 38.2 | — | — | — | |
| HY-1.8B-2BitModel Scale=1.8B, Precision=2Bit, Instruction-tuned=true2026.02 | 32.73 | — | — | — | |
| AZR-R-TAPBase Model Type=Base2026.03 | 32.4 | — | — | — | |
| HY-1.8B-FP16Model Scale=1.8B, Precision=FP16, Instruction-tuned=true2026.02 | 31.5 | — | — | — | |
| HY-1.8B-INT4Model Scale=1.8B, Precision=INT4, Instruction-tuned=true2026.02 | 30.08 | — | — | — | |
| HY-0.5B-FP16Model Scale=0.5B, Precision=FP16, Instruction-tuned=true2026.02 | 12.11 | — | — | — |