Code Correctness Prediction on LiveCodeBench Python (AUROC)
86.7AUROCFunc. Equiv.
Evaluation Results
| Method | Links | |
|---|---|---|
| Func. Equiv.Model=Gemini-2.5-Flash, Category=Functional Equivalence Methods2026.05 | 86.7 | |
| Func. Equiv.Model=Gemini-2.5-Flash-Lite, Category=Functional Equivalence Methods2026.05 | 85.9 | |
| Func. Equiv.Model=GPT-4o-mini, Category=Functional Equivalence Methods2026.05 | 85.9 | |
| Func. EntropyModel=GPT-4o-mini, Category=Functional Equivalence Methods2026.05 | 85.9 | |
| Func. EntropyModel=Gemini-2.5-Flash, Category=Functional Equivalence Methods2026.05 | 85.7 | |
| Func. EntropyModel=Gemini-2.5-Flash-Lite, Category=Functional Equivalence Methods2026.05 | 85.7 | |
| Func. Equiv.Model=GPT-4o, Category=Functional Equivalence Methods2026.05 | 85.6 | |
| Func. SetsModel=GPT-4o-mini, Category=Functional Equivalence Methods2026.05 | 85.6 | |
| Func. EntropyModel=GPT-4o, Category=Functional Equivalence Methods2026.05 | 85.5 | |
| Func. SetsModel=Gemini-2.5-Flash, Category=Functional Equivalence Methods2026.05 | 85.5 | |
| Func. SetsModel=GPT-4o, Category=Functional Equivalence Methods2026.05 | 85.5 | |
| Func. SetsModel=Gemini-2.5-Flash-Lite, Category=Functional Equivalence Methods2026.05 | 85.2 | |
| P(True)Model=Gemini-2.5-Flash-Lite, Category=Reflexive Methods2026.05 | 83 | |
| Func. EntropyModel=Gemini-2.5-Pro, Category=Functional Equivalence Methods2026.05 | 82.9 | |
| Func. Equiv.Model=Gemini-2.5-Pro, Category=Functional Equivalence Methods2026.05 | 82.6 | |
| Func. SetsModel=Gemini-2.5-Pro, Category=Functional Equivalence Methods2026.05 | 82.6 | |
| P(True)Model=GPT-4o, Category=Reflexive Methods2026.05 | 82.4 | |
| P(True)Model=Gemini-2.5-Pro, Category=Reflexive Methods2026.05 | 81.5 | |
| CodeBLEUModel=Gemini-2.5-Pro, Category=Similarity-Based Methods2026.05 | 81.3 | |
| Cos. Sim.Model=Gemini-2.5-Pro, Category=Similarity-Based Methods2026.05 | 80.4 | |
| Min. Tok. Neg.Model=Gemini-2.5-Flash-Lite, Category=Token-Probability Methods2026.05 | 79.7 | |
| CodeBLEUModel=Gemini-2.5-Flash, Category=Similarity-Based Methods2026.05 | 79.5 | |
| CodeBLEUModel=Gemini-2.5-Flash-Lite, Category=Similarity-Based Methods2026.05 | 79.5 | |
| Verb. Conf.Model=Gemini-2.5-Flash, Category=Reflexive Methods2026.05 | 79.5 | |
| Min. Prob.Model=Gemini-2.5-Flash-Lite, Category=Token-Probability Methods2026.05 | 78.3 | |
| Min. Tok. Neg.Model=GPT-4o-mini, Category=Token-Probability Methods2026.05 | 77.9 | |
| Avg. Tok. Neg.Model=GPT-4o-mini, Category=Token-Probability Methods2026.05 | 77.6 | |
| Avg. Tok. Neg.Model=Gemini-2.5-Flash-Lite, Category=Token-Probability Methods2026.05 | 77.4 | |
| Min. Prob.Model=Gemini-2.5-Flash, Category=Token-Probability Methods2026.05 | 77.3 | |
| P(True)Model=Gemini-2.5-Flash, Category=Reflexive Methods2026.05 | 77.1 | |
| Min. Tok. Neg.Model=Gemini-2.5-Pro, Category=Token-Probability Methods2026.05 | 77 | |
| Min. Tok. Neg.Model=GPT-4o, Category=Token-Probability Methods2026.05 | 76.8 | |
| Min. Tok. Neg.Model=Gemini-2.5-Flash, Category=Token-Probability Methods2026.05 | 76.6 | |
| CodeBLEUModel=GPT-4o-mini, Category=Similarity-Based Methods2026.05 | 76.6 | |
| Min. Prob.Model=Gemini-2.5-Pro, Category=Token-Probability Methods2026.05 | 76.2 | |
| Prob. MarginModel=Gemini-2.5-Flash-Lite, Category=Token-Probability Methods2026.05 | 76.2 | |
| Seq. Prob.Model=Gemini-2.5-Flash-Lite, Category=Token-Probability Methods2026.05 | 76.1 | |
| Avg. Tok. Neg.Model=GPT-4o, Category=Token-Probability Methods2026.05 | 76 | |
| Seq. Prob.Model=GPT-4o-mini, Category=Token-Probability Methods2026.05 | 75.7 | |
| Cos. Sim.Model=Gemini-2.5-Flash-Lite, Category=Similarity-Based Methods2026.05 | 75.6 | |
| Cos. Sim.Model=Gemini-2.5-Flash, Category=Similarity-Based Methods2026.05 | 75.3 | |
| Prob. MarginModel=GPT-4o-mini, Category=Token-Probability Methods2026.05 | 75.2 | |
| P(True)Model=GPT-4o-mini, Category=Reflexive Methods2026.05 | 75.2 | |
| Prob. MarginModel=GPT-4o, Category=Token-Probability Methods2026.05 | 74.9 | |
| CodeBLEUModel=GPT-4o, Category=Similarity-Based Methods2026.05 | 74.4 | |
| Min. Prob.Model=GPT-4o-mini, Category=Token-Probability Methods2026.05 | 73.8 | |
| Cos. Sim.Model=GPT-4o-mini, Category=Similarity-Based Methods2026.05 | 73.8 | |
| Seq. Prob.Model=GPT-4o, Category=Token-Probability Methods2026.05 | 72.9 | |
| Verb. Conf.Model=GPT-4o-mini, Category=Reflexive Methods2026.05 | 72.5 | |
| Cos. Sim.Model=GPT-4o, Category=Similarity-Based Methods2026.05 | 72.4 | |
| Verb. Conf.Model=Gemini-2.5-Pro, Category=Reflexive Methods2026.05 | 71.6 | |
| Min. Prob.Model=GPT-4o, Category=Token-Probability Methods2026.05 | 70.5 | |
| Avg. Tok. Neg.Model=Gemini-2.5-Flash, Category=Token-Probability Methods2026.05 | 69.5 | |
| Seq. Prob.Model=Gemini-2.5-Flash, Category=Token-Probability Methods2026.05 | 69.1 | |
| Verb. Conf.Model=GPT-4o, Category=Reflexive Methods2026.05 | 68.1 | |
| Prob. MarginModel=Gemini-2.5-Flash, Category=Token-Probability Methods2026.05 | 66.8 | |
| Verb. Conf.Model=Gemini-2.5-Flash-Lite, Category=Reflexive Methods2026.05 | 59.7 | |
| Avg. Tok. Neg.Model=Gemini-2.5-Pro, Category=Token-Probability Methods2026.05 | 58.2 | |
| Seq. Prob.Model=Gemini-2.5-Pro, Category=Token-Probability Methods2026.05 | 58 | |
| Prob. MarginModel=Gemini-2.5-Pro, Category=Token-Probability Methods2026.05 | 55.4 |