Code Generation on LiveCodeBench (Pass@1, Pass@10)
88.1Pass@1SEP
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SEPModel=gpt-oss-20b, N=102026.04 | 88.1 | — | |
| CodeTModel=gpt-oss-20b, N=102026.04 | 87.8 | — | |
| SEPModel=Phi-4-reasoning, N=102026.04 | 86.3 | — | |
| Embedding ConsensusModel=gpt-oss-20b, N=102026.04 | 86.1 | — | |
| Embedding ConsensusModel=Qwen3-14B, N=102026.04 | 85.8 | — | |
| SEPModel=Qwen3-14B, N=102026.04 | 85.4 | — | |
| LLM-as-a-JudgeModel=gpt-oss-20b, N=102026.04 | 84.3 | — | |
| CodeBLEU ConsensusModel=gpt-oss-20b, N=102026.04 | 84 | — | |
| CodeTModel=Phi-4-reasoning, N=102026.04 | 83.4 | — | |
| CodeTModel=Qwen3-14B, N=102026.04 | 83.2 | — | |
| LLM-as-a-JudgeModel=Qwen3-14B, N=102026.04 | 83.1 | — | |
| UnguidedModel=Qwen3-14B, N=102026.04 | 83 | — | |
| UnguidedModel=gpt-oss-20b, N=102026.04 | 82.6 | — | |
| CodeBLEU ConsensusModel=Qwen3-14B, N=102026.04 | 82 | — | |
| SEPModel=Qwen3-4B, N=102026.04 | 79.5 | — | |
| Embedding ConsensusModel=Qwen3-4B, N=102026.04 | 77.2 | — | |
| CodeTModel=Qwen3-4B, N=102026.04 | 75.9 | — | |
| Embedding ConsensusModel=Phi-4-reasoning, N=102026.04 | 75.8 | — | |
| UnguidedModel=Qwen3-4B, N=102026.04 | 74.9 | — | |
| CodeBLEU ConsensusModel=Phi-4-reasoning, N=102026.04 | 74.4 | — | |
| CodeBLEU ConsensusModel=Qwen3-4B, N=102026.04 | 74.4 | — | |
| UnguidedModel=Phi-4-reasoning, N=102026.04 | 73.3 | — | |
| LLM-as-a-JudgeModel=Qwen3-4B, N=102026.04 | 73.1 | — | |
| LLM-as-a-JudgeModel=Phi-4-reasoning, N=102026.04 | 72.6 | — | |
| SEPModel=Qwen3-4B-Instruct-2507, N=102026.04 | 62.1 | — | |
| FP16Model=Qwen3-8B, Calibration Dataset=N/A, Quantization Bit=N/A, Group Size=N/A2026.05 | 58.46 | — | |
| CodeTModel=Qwen3-4B-Instruct-2507, N=102026.04 | 57.9 | — | |
| LAQuantModel=Qwen3-8B, Calibration Dataset=OpenR1, Quantization Bit=4, Group Size=1282026.05 | 56.59 | — | |
| ParoQuant++Model=Qwen3-8B, Calibration Dataset=OpenR1, Quantization Bit=4, Group Size=1282026.05 | 55.72 | — | |
| LLM-as-a-JudgeModel=Qwen3-4B-Instruct-2507, N=102026.04 | 55.7 | — | |
| ParoQuantModel=Qwen3-8B, Calibration Dataset=Generic, Quantization Bit=4, Group Size=1282026.05 | 55.47 | — | |
| GPTQ++Model=Qwen3-8B, Calibration Dataset=OpenR1, Quantization Bit=4, Group Size=1282026.05 | 55.1 | — | |
| FP16Model=Qwen3-4B, Calibration Dataset=N/A, Quantization Bit=N/A, Group Size=N/A2026.05 | 54.48 | — | |
| Embedding ConsensusModel=Qwen3-4B-Instruct-2507, N=102026.04 | 54.3 | — | |
| CodeBLEU ConsensusModel=Qwen3-4B-Instruct-2507, N=102026.04 | 53.4 | — | |
| GPTQModel=Qwen3-8B, Calibration Dataset=Generic, Quantization Bit=4, Group Size=1282026.05 | 52.99 | — | |
| LAQuantModel=Qwen3-4B, Calibration Dataset=OpenR1, Quantization Bit=4, Group Size=1282026.05 | 52.74 | — | |
| UnguidedModel=Qwen3-4B-Instruct-2507, N=102026.04 | 52.1 | — | |
| ParoQuant++Model=Qwen3-4B, Calibration Dataset=OpenR1, Quantization Bit=4, Group Size=1282026.05 | 51.62 | — | |
| ReasoningQATModel=Qwen3-8B, Calibration Dataset=Mixed, Quantization Bit=4, Group Size=1282026.05 | 51.62 | — | |
| GPTQ++Model=Qwen3-4B, Calibration Dataset=OpenR1, Quantization Bit=4, Group Size=1282026.05 | 50.75 | — | |
| ParoQuantModel=Qwen3-4B, Calibration Dataset=Generic, Quantization Bit=4, Group Size=1282026.05 | 50 | — | |
| GPTQModel=Qwen3-4B, Calibration Dataset=Generic, Quantization Bit=4, Group Size=1282026.05 | 48.75 | — | |
| ReasoningQATModel=Qwen3-4B, Calibration Dataset=Mixed, Quantization Bit=4, Group Size=1282026.05 | 46.76 | — | |
| FP16Model=R1-Distill-Llama-8B, Calibration Dataset=N/A, Quantization Bit=N/A, Group Size=N/A2026.05 | 36.32 | — | |
| LAQuantModel=R1-Distill-Llama-8B, Calibration Dataset=OpenR1, Quantization Bit=4, Group Size=1282026.05 | 35.7 | — | |
| GPTQ++Model=R1-Distill-Llama-8B, Calibration Dataset=OpenR1, Quantization Bit=4, Group Size=1282026.05 | 34.45 | — | |
| ReasoningQATModel=R1-Distill-Llama-8B, Calibration Dataset=Mixed, Quantization Bit=4, Group Size=1282026.05 | 34.33 | — | |
| GPTQModel=R1-Distill-Llama-8B, Calibration Dataset=Generic, Quantization Bit=4, Group Size=1282026.05 | 34.21 | — | |
| FP16Model=Qwen3-1.7B, Calibration Dataset=N/A, Quantization Bit=N/A, Group Size=N/A2026.05 | 33.71 | — | |
| ParoQuantModel=R1-Distill-Llama-8B, Calibration Dataset=Generic, Quantization Bit=4, Group Size=1282026.05 | 33.58 | — | |
| ParoQuant++Model=R1-Distill-Llama-8B, Calibration Dataset=OpenR1, Quantization Bit=4, Group Size=1282026.05 | 33.58 | — | |
| LAQuantModel=Qwen3-1.7B, Calibration Dataset=OpenR1, Quantization Bit=4, Group Size=1282026.05 | 31.84 | — | |
| SEPModel=Phi-4-mini-reasoning, N=102026.04 | 30.8 | — | |
| ParoQuant++Model=Qwen3-1.7B, Calibration Dataset=OpenR1, Quantization Bit=4, Group Size=1282026.05 | 28.98 | — | |
| ParoQuantModel=Qwen3-1.7B, Calibration Dataset=Generic, Quantization Bit=4, Group Size=1282026.05 | 28.61 | — | |
| GPTQ++Model=Qwen3-1.7B, Calibration Dataset=OpenR1, Quantization Bit=4, Group Size=1282026.05 | 28.48 | — | |
| CodeTModel=Phi-4-mini-reasoning, N=102026.04 | 26.2 | — | |
| ReasoningQATModel=Qwen3-1.7B, Calibration Dataset=Mixed, Quantization Bit=4, Group Size=1282026.05 | 25.87 | — | |
| SEPModel=Llama-3.1-8B-Instruct, N=102026.04 | 25.6 | — | |
| SEPModel=Phi-4-mini-instruct, N=102026.04 | 25.6 | — | |
| CodeBLEU ConsensusModel=Phi-4-mini-reasoning, N=102026.04 | 23.7 | — | |
| CodeTModel=Phi-4-mini-instruct, N=102026.04 | 21.3 | — | |
| GPTQModel=Qwen3-1.7B, Calibration Dataset=Generic, Quantization Bit=4, Group Size=1282026.05 | 20.4 | — | |
| CodeBLEU ConsensusModel=Phi-4-mini-instruct, N=102026.04 | 18.5 | — | |
| LLM-as-a-JudgeModel=Phi-4-mini-reasoning, N=102026.04 | 18.3 | — | |
| CodeTModel=Llama-3.1-8B-Instruct, N=102026.04 | 18 | — | |
| Embedding ConsensusModel=Phi-4-mini-instruct, N=102026.04 | 17.8 | — | |
| LLM-as-a-JudgeModel=Phi-4-mini-instruct, N=102026.04 | 17.6 | — | |
| UnguidedModel=Phi-4-mini-reasoning, N=102026.04 | 17.6 | — | |
| Embedding ConsensusModel=Phi-4-mini-reasoning, N=102026.04 | 17.4 | — | |
| UnguidedModel=Phi-4-mini-instruct, N=102026.04 | 16.9 | — | |
| CodeBLEU ConsensusModel=Llama-3.1-8B-Instruct, N=102026.04 | 14.8 | — | |
| Embedding ConsensusModel=Llama-3.1-8B-Instruct, N=102026.04 | 13.7 | — | |
| UnguidedModel=Llama-3.1-8B-Instruct, N=102026.04 | 12.7 | — | |
| LLM-as-a-JudgeModel=Llama-3.1-8B-Instruct, N=102026.04 | 11.9 | — | |
| Oracle CeilingModel=Llama-3.1-8B-Instruct, N=102026.04 | — | 27.4 | |
| Oracle CeilingModel=Phi-4-mini-instruct, N=102026.04 | — | 29 | |
| Oracle CeilingModel=Phi-4-mini-reasoning, N=102026.04 | — | 31.1 | |
| Oracle CeilingModel=Phi-4-reasoning, N=102026.04 | — | 93.2 | |
| Oracle CeilingModel=Qwen3-4B-Instruct-2507, N=102026.04 | — | 69.2 | |
| Oracle CeilingModel=Qwen3-4B, N=102026.04 | — | 85.4 | |
| Oracle CeilingModel=Qwen3-14B, N=102026.04 | — | 92.2 | |
| Oracle CeilingModel=gpt-oss-20b, N=102026.04 | — | 96.6 |