Code Correctness Prediction on MultiPL-E Java (AUROC)
0.705AUROCMin. Prob.
Evaluation Results
| Method | Links | |
|---|---|---|
| Min. Prob.Model=Gemini-2.5-Flash-Lite2026.05 | 0.705 | |
| Func. EntropyModel=Gemini-2.5-Flash-Lite2026.05 | 0.701 | |
| Func. SetsModel=Gemini-2.5-Flash-Lite2026.05 | 0.697 | |
| Func. EntropyModel=GPT-4o-mini2026.05 | 0.695 | |
| Func. SetsModel=GPT-4o-mini2026.05 | 0.693 | |
| Func. SetsModel=GPT-4o2026.05 | 0.691 | |
| Func. Equiv.Model=Gemini-2.5-Flash-Lite2026.05 | 0.69 | |
| CodeBLEUModel=Gemini-2.5-Flash2026.05 | 0.69 | |
| Func. Equiv.Model=GPT-4o2026.05 | 0.687 | |
| P(True)Model=GPT-4o2026.05 | 0.687 | |
| Func. EntropyModel=GPT-4o2026.05 | 0.686 | |
| Func. Equiv.Model=GPT-4o-mini2026.05 | 0.681 | |
| CodeBLEUModel=Gemini-2.5-Flash-Lite2026.05 | 0.676 | |
| Min. Tok. Neg.Model=Gemini-2.5-Flash-Lite2026.05 | 0.674 | |
| CodeBLEUModel=GPT-4o2026.05 | 0.672 | |
| Func. Equiv.Model=Gemini-2.5-Pro2026.05 | 0.665 | |
| Func. SetsModel=Gemini-2.5-Pro2026.05 | 0.663 | |
| Func. EntropyModel=Gemini-2.5-Pro2026.05 | 0.662 | |
| CodeBLEUModel=Gemini-2.5-Pro2026.05 | 0.662 | |
| CodeBLEUModel=GPT-4o-mini2026.05 | 0.658 | |
| Min. Tok. Neg.Model=GPT-4o2026.05 | 0.653 | |
| Min. Tok. Neg.Model=Gemini-2.5-Flash2026.05 | 0.652 | |
| Min. Prob.Model=GPT-4o2026.05 | 0.647 | |
| Cos. Sim.Model=Gemini-2.5-Flash2026.05 | 0.645 | |
| Avg. Tok. Neg.Model=GPT-4o2026.05 | 0.644 | |
| Min. Tok. Neg.Model=Gemini-2.5-Pro2026.05 | 0.644 | |
| Prob. MarginModel=GPT-4o2026.05 | 0.64 | |
| P(True)Model=GPT-4o-mini2026.05 | 0.64 | |
| Seq. Prob.Model=GPT-4o2026.05 | 0.638 | |
| Seq. Prob.Model=Gemini-2.5-Flash-Lite2026.05 | 0.637 | |
| Min. Prob.Model=Gemini-2.5-Flash2026.05 | 0.635 | |
| Min. Tok. Neg.Model=GPT-4o-mini2026.05 | 0.632 | |
| Func. EntropyModel=Gemini-2.5-Flash2026.05 | 0.628 | |
| Cos. Sim.Model=Gemini-2.5-Pro2026.05 | 0.627 | |
| Cos. Sim.Model=GPT-4o2026.05 | 0.626 | |
| Func. SetsModel=Gemini-2.5-Flash2026.05 | 0.623 | |
| Verb. Conf.Model=Gemini-2.5-Flash2026.05 | 0.623 | |
| Min. Prob.Model=Gemini-2.5-Pro2026.05 | 0.618 | |
| Func. Equiv.Model=Gemini-2.5-Flash2026.05 | 0.616 | |
| P(True)Model=Gemini-2.5-Pro2026.05 | 0.616 | |
| Cos. Sim.Model=Gemini-2.5-Flash-Lite2026.05 | 0.615 | |
| Verb. Conf.Model=GPT-4o-mini2026.05 | 0.615 | |
| Cos. Sim.Model=GPT-4o-mini2026.05 | 0.61 | |
| Prob. MarginModel=Gemini-2.5-Flash-Lite2026.05 | 0.601 | |
| Min. Prob.Model=GPT-4o-mini2026.05 | 0.599 | |
| Avg. Tok. Neg.Model=Gemini-2.5-Pro2026.05 | 0.598 | |
| Prob. MarginModel=Gemini-2.5-Pro2026.05 | 0.596 | |
| Verb. Conf.Model=Gemini-2.5-Flash-Lite2026.05 | 0.595 | |
| Verb. Conf.Model=GPT-4o2026.05 | 0.592 | |
| Avg. Tok. Neg.Model=Gemini-2.5-Flash-Lite2026.05 | 0.589 | |
| Seq. Prob.Model=Gemini-2.5-Pro2026.05 | 0.588 | |
| Seq. Prob.Model=GPT-4o-mini2026.05 | 0.572 | |
| Prob. MarginModel=GPT-4o-mini2026.05 | 0.553 | |
| Seq. Prob.Model=Gemini-2.5-Flash2026.05 | 0.551 | |
| Avg. Tok. Neg.Model=GPT-4o-mini2026.05 | 0.551 | |
| Verb. Conf.Model=Gemini-2.5-Pro2026.05 | 0.548 | |
| P(True)Model=Gemini-2.5-Flash-Lite2026.05 | 0.548 | |
| Avg. Tok. Neg.Model=Gemini-2.5-Flash2026.05 | 0.536 | |
| P(True)Model=Gemini-2.5-Flash2026.05 | 0.529 | |
| Prob. MarginModel=Gemini-2.5-Flash2026.05 | 0.525 |