List your company
ResearchFormerly WizWand

Code Correctness Prediction on MultiPL-E Java (AUROC)

0.705AUROCMin. Prob.
0.51780.56640.6150.6636May 27, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Min. Prob.Model=Gemini-2.5-Flash-Lite2026.050.705
Func. EntropyModel=Gemini-2.5-Flash-Lite2026.050.701
Func. SetsModel=Gemini-2.5-Flash-Lite2026.050.697
Func. EntropyModel=GPT-4o-mini2026.050.695
Func. SetsModel=GPT-4o-mini2026.050.693
Func. SetsModel=GPT-4o2026.050.691
Func. Equiv.Model=Gemini-2.5-Flash-Lite2026.050.69
CodeBLEUModel=Gemini-2.5-Flash2026.050.69
Func. Equiv.Model=GPT-4o2026.050.687
P(True)Model=GPT-4o2026.050.687
Func. EntropyModel=GPT-4o2026.050.686
Func. Equiv.Model=GPT-4o-mini2026.050.681
CodeBLEUModel=Gemini-2.5-Flash-Lite2026.050.676
Min. Tok. Neg.Model=Gemini-2.5-Flash-Lite2026.050.674
CodeBLEUModel=GPT-4o2026.050.672
Func. Equiv.Model=Gemini-2.5-Pro2026.050.665
Func. SetsModel=Gemini-2.5-Pro2026.050.663
Func. EntropyModel=Gemini-2.5-Pro2026.050.662
CodeBLEUModel=Gemini-2.5-Pro2026.050.662
CodeBLEUModel=GPT-4o-mini2026.050.658
Min. Tok. Neg.Model=GPT-4o2026.050.653
Min. Tok. Neg.Model=Gemini-2.5-Flash2026.050.652
Min. Prob.Model=GPT-4o2026.050.647
Cos. Sim.Model=Gemini-2.5-Flash2026.050.645
Avg. Tok. Neg.Model=GPT-4o2026.050.644
Min. Tok. Neg.Model=Gemini-2.5-Pro2026.050.644
Prob. MarginModel=GPT-4o2026.050.64
P(True)Model=GPT-4o-mini2026.050.64
Seq. Prob.Model=GPT-4o2026.050.638
Seq. Prob.Model=Gemini-2.5-Flash-Lite2026.050.637
Min. Prob.Model=Gemini-2.5-Flash2026.050.635
Min. Tok. Neg.Model=GPT-4o-mini2026.050.632
Func. EntropyModel=Gemini-2.5-Flash2026.050.628
Cos. Sim.Model=Gemini-2.5-Pro2026.050.627
Cos. Sim.Model=GPT-4o2026.050.626
Func. SetsModel=Gemini-2.5-Flash2026.050.623
Verb. Conf.Model=Gemini-2.5-Flash2026.050.623
Min. Prob.Model=Gemini-2.5-Pro2026.050.618
Func. Equiv.Model=Gemini-2.5-Flash2026.050.616
P(True)Model=Gemini-2.5-Pro2026.050.616
Cos. Sim.Model=Gemini-2.5-Flash-Lite2026.050.615
Verb. Conf.Model=GPT-4o-mini2026.050.615
Cos. Sim.Model=GPT-4o-mini2026.050.61
Prob. MarginModel=Gemini-2.5-Flash-Lite2026.050.601
Min. Prob.Model=GPT-4o-mini2026.050.599
Avg. Tok. Neg.Model=Gemini-2.5-Pro2026.050.598
Prob. MarginModel=Gemini-2.5-Pro2026.050.596
Verb. Conf.Model=Gemini-2.5-Flash-Lite2026.050.595
Verb. Conf.Model=GPT-4o2026.050.592
Avg. Tok. Neg.Model=Gemini-2.5-Flash-Lite2026.050.589
Seq. Prob.Model=Gemini-2.5-Pro2026.050.588
Seq. Prob.Model=GPT-4o-mini2026.050.572
Prob. MarginModel=GPT-4o-mini2026.050.553
Seq. Prob.Model=Gemini-2.5-Flash2026.050.551
Avg. Tok. Neg.Model=GPT-4o-mini2026.050.551
Verb. Conf.Model=Gemini-2.5-Pro2026.050.548
P(True)Model=Gemini-2.5-Flash-Lite2026.050.548
Avg. Tok. Neg.Model=Gemini-2.5-Flash2026.050.536
P(True)Model=Gemini-2.5-Flash2026.050.529
Prob. MarginModel=Gemini-2.5-Flash2026.050.525