List your company
ResearchFormerly WizWand

Predicting code correctness on LiveSQLBench SQLite

0.129Brier ScoreFunc. Neg.
0.10480.268150.43150.59485May 27, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Func. Neg.Model=GPT4o-m, Scorer Category=Functional Equivalence Methods2026.050.129
Func. SetsModel=GPT4o-m, Scorer Category=Functional Equivalence Methods2026.050.135
Func. Equiv. RateModel=GPT4o-m, Scorer Category=Functional Equivalence Methods2026.050.147
Func. Neg.Model=Gem-Fl-Lt, Scorer Category=Functional Equivalence Methods2026.050.168
Func. Equiv. RateModel=Gem-Fl-Lt, Scorer Category=Functional Equivalence Methods2026.050.169
Func. SetsModel=GPT4o, Scorer Category=Functional Equivalence Methods2026.050.174
Min. Prob.Model=GPT4o-m, Scorer Category=Token-Probability Methods2026.050.175
Func. SetsModel=Gem-Fl-Lt, Scorer Category=Functional Equivalence Methods2026.050.176
Func. Neg.Model=GPT4o, Scorer Category=Functional Equivalence Methods2026.050.183
Min. Tok. Neg.Model=GPT4o, Scorer Category=Token-Probability Methods2026.050.184
Min. Prob.Model=Gem-Fl-Lt, Scorer Category=Token-Probability Methods2026.050.199
Min. Tok. Neg.Model=Gem-Fl-Lt, Scorer Category=Token-Probability Methods2026.050.2
Func. Equiv. RateModel=GPT4o, Scorer Category=Functional Equivalence Methods2026.050.2
Func. SetsModel=Gem-Pro, Scorer Category=Functional Equivalence Methods2026.050.207
Func. Neg.Model=Gem-Pro, Scorer Category=Functional Equivalence Methods2026.050.215
Min. Tok. Neg.Model=GPT4o-m, Scorer Category=Token-Probability Methods2026.050.225
Min. Tok. Neg.Model=Gem-Fl, Scorer Category=Token-Probability Methods2026.050.248
Func. SetsModel=Gem-Fl, Scorer Category=Functional Equivalence Methods2026.050.248
Func. Neg.Model=Gem-Fl, Scorer Category=Functional Equivalence Methods2026.050.261
Func. Equiv. RateModel=Gem-Pro, Scorer Category=Functional Equivalence Methods2026.050.263
P(True)Model=GPT4o, Scorer Category=Reflexive Methods2026.050.264
Min. Prob.Model=GPT4o, Scorer Category=Token-Probability Methods2026.050.266
Min. Tok. Neg.Model=Gem-Pro, Scorer Category=Token-Probability Methods2026.050.266
Min. Prob.Model=Gem-Fl, Scorer Category=Token-Probability Methods2026.050.302
Func. Equiv. RateModel=Gem-Fl, Scorer Category=Functional Equivalence Methods2026.050.302
Min. Prob.Model=Gem-Pro, Scorer Category=Token-Probability Methods2026.050.308
Verb. Conf.Model=Gem-Fl-Lt, Scorer Category=Reflexive Methods2026.050.368
P(True)Model=Gem-Fl-Lt, Scorer Category=Reflexive Methods2026.050.381
Seq. Prob.Model=GPT4o, Scorer Category=Token-Probability Methods2026.050.398
P(True)Model=Gem-Pro, Scorer Category=Reflexive Methods2026.050.422
Verb. Conf.Model=Gem-Fl, Scorer Category=Reflexive Methods2026.050.43
Verb. Conf.Model=GPT4o-m, Scorer Category=Reflexive Methods2026.050.431
Verb. Conf.Model=Gem-Pro, Scorer Category=Reflexive Methods2026.050.46
P(True)Model=GPT4o-m, Scorer Category=Reflexive Methods2026.050.477
P(True)Model=Gem-Fl, Scorer Category=Reflexive Methods2026.050.485
Cos. Sim.Model=Gem-Pro, Scorer Category=Similarity-Based Methods2026.050.503
Cos. Sim.Model=Gem-Fl, Scorer Category=Similarity-Based Methods2026.050.51
Prob. MarginModel=Gem-Pro, Scorer Category=Token-Probability Methods2026.050.518
Seq. Prob.Model=Gem-Pro, Scorer Category=Token-Probability Methods2026.050.519
Seq. Prob.Model=Gem-Fl, Scorer Category=Token-Probability Methods2026.050.52
Prob. MarginModel=Gem-Fl, Scorer Category=Token-Probability Methods2026.050.521
Avg. Tok. Neg.Model=Gem-Fl, Scorer Category=Token-Probability Methods2026.050.532
Avg. Tok. Neg.Model=Gem-Pro, Scorer Category=Token-Probability Methods2026.050.532
Verb. Conf.Model=GPT4o, Scorer Category=Reflexive Methods2026.050.548
Prob. MarginModel=GPT4o, Scorer Category=Token-Probability Methods2026.050.553
Avg. Tok. Neg.Model=GPT4o, Scorer Category=Token-Probability Methods2026.050.604
Cos. Sim.Model=GPT4o, Scorer Category=Similarity-Based Methods2026.050.62
Seq. Prob.Model=GPT4o-m, Scorer Category=Token-Probability Methods2026.050.627
Prob. MarginModel=GPT4o-m, Scorer Category=Token-Probability Methods2026.050.686
Seq. Prob.Model=Gem-Fl-Lt, Scorer Category=Token-Probability Methods2026.050.687
Prob. MarginModel=Gem-Fl-Lt, Scorer Category=Token-Probability Methods2026.050.692
Cos. Sim.Model=Gem-Fl-Lt, Scorer Category=Similarity-Based Methods2026.050.705
Avg. Tok. Neg.Model=Gem-Fl-Lt, Scorer Category=Token-Probability Methods2026.050.724
Avg. Tok. Neg.Model=GPT4o-m, Scorer Category=Token-Probability Methods2026.050.729
Cos. Sim.Model=GPT4o-m, Scorer Category=Similarity-Based Methods2026.050.734