Code Correctness Prediction on MultiPL-E Java (Brier Score)
0.231Brier ScoreCodeBLEU
Evaluation Results
| Method | Links | |
|---|---|---|
| CodeBLEUModel=Gem-Fl2026.05 | 0.231 | |
| Min. Tok. Neg.Model=Gem-Fl-Lt2026.05 | 0.232 | |
| CodeBLEUModel=GPT4o2026.05 | 0.233 | |
| CodeBLEUModel=Gem-Pro2026.05 | 0.236 | |
| Min. Tok. Neg.Model=GPT4o2026.05 | 0.237 | |
| Func. Neg.Model=GPT4o2026.05 | 0.243 | |
| Min. Tok. Neg.Model=GPT4o-m2026.05 | 0.245 | |
| Min. Tok. Neg.Model=Gem-Fl2026.05 | 0.25 | |
| CodeBLEUModel=GPT4o-m2026.05 | 0.252 | |
| Func. Neg.Model=GPT4o-m2026.05 | 0.254 | |
| Func. Equiv. RateModel=GPT4o2026.05 | 0.259 | |
| Min. Prob.Model=Gem-Fl-Lt2026.05 | 0.263 | |
| Min. Tok. Neg.Model=Gem-Pro2026.05 | 0.263 | |
| Func. SetsModel=GPT4o2026.05 | 0.266 | |
| CodeBLEUModel=Gem-Fl-Lt2026.05 | 0.271 | |
| Func. Neg.Model=Gem-Fl-Lt2026.05 | 0.272 | |
| Func. Equiv. RateModel=GPT4o-m2026.05 | 0.273 | |
| Func. Equiv. RateModel=Gem-Fl-Lt2026.05 | 0.274 | |
| Func. Neg.Model=Gem-Pro2026.05 | 0.275 | |
| Func. Equiv. RateModel=Gem-Pro2026.05 | 0.28 | |
| Verb. Conf.Model=GPT4o-m2026.05 | 0.282 | |
| Func. SetsModel=GPT4o-m2026.05 | 0.283 | |
| Func. Neg.Model=Gem-Fl2026.05 | 0.284 | |
| Min. Prob.Model=Gem-Pro2026.05 | 0.303 | |
| Func. Equiv. RateModel=Gem-Fl2026.05 | 0.303 | |
| Min. Prob.Model=Gem-Fl2026.05 | 0.306 | |
| Func. SetsModel=Gem-Pro2026.05 | 0.314 | |
| Func. SetsModel=Gem-Fl-Lt2026.05 | 0.314 | |
| Min. Prob.Model=GPT4o2026.05 | 0.325 | |
| Func. SetsModel=Gem-Fl2026.05 | 0.326 | |
| P(True)Model=GPT4o2026.05 | 0.331 | |
| Min. Prob.Model=GPT4o-m2026.05 | 0.34 | |
| Verb. Conf.Model=GPT4o2026.05 | 0.354 | |
| P(True)Model=GPT4o-m2026.05 | 0.36 | |
| Seq. Prob.Model=GPT4o2026.05 | 0.364 | |
| Prob. MarginModel=GPT4o2026.05 | 0.378 | |
| Verb. Conf.Model=Gem-Fl2026.05 | 0.379 | |
| Seq. Prob.Model=GPT4o-m2026.05 | 0.383 | |
| Prob. MarginModel=GPT4o-m2026.05 | 0.394 | |
| Cos. Sim.Model=GPT4o2026.05 | 0.396 | |
| Avg. Tok. Neg.Model=GPT4o2026.05 | 0.4 | |
| Cos. Sim.Model=GPT4o-m2026.05 | 0.413 | |
| Avg. Tok. Neg.Model=GPT4o-m2026.05 | 0.415 | |
| Cos. Sim.Model=Gem-Pro2026.05 | 0.419 | |
| P(True)Model=Gem-Fl-Lt2026.05 | 0.419 | |
| P(True)Model=Gem-Pro2026.05 | 0.423 | |
| Verb. Conf.Model=Gem-Pro2026.05 | 0.425 | |
| Prob. MarginModel=Gem-Pro2026.05 | 0.432 | |
| Seq. Prob.Model=Gem-Fl2026.05 | 0.434 | |
| Seq. Prob.Model=Gem-Pro2026.05 | 0.435 | |
| Prob. MarginModel=Gem-Fl2026.05 | 0.438 | |
| Verb. Conf.Model=Gem-Fl-Lt2026.05 | 0.438 | |
| Cos. Sim.Model=Gem-Fl2026.05 | 0.445 | |
| Avg. Tok. Neg.Model=Gem-Pro2026.05 | 0.45 | |
| Seq. Prob.Model=Gem-Fl-Lt2026.05 | 0.46 | |
| Avg. Tok. Neg.Model=Gem-Fl2026.05 | 0.463 | |
| Prob. MarginModel=Gem-Fl-Lt2026.05 | 0.465 | |
| P(True)Model=Gem-Fl2026.05 | 0.466 | |
| Cos. Sim.Model=Gem-Fl-Lt2026.05 | 0.484 | |
| Avg. Tok. Neg.Model=Gem-Fl-Lt2026.05 | 0.49 |