Predicting code correctness on LiveSQLBench SQLite
0.129Brier ScoreFunc. Neg.
Evaluation Results
| Method | Links | |
|---|---|---|
| Func. Neg.Model=GPT4o-m, Scorer Category=Functional Equivalence Methods2026.05 | 0.129 | |
| Func. SetsModel=GPT4o-m, Scorer Category=Functional Equivalence Methods2026.05 | 0.135 | |
| Func. Equiv. RateModel=GPT4o-m, Scorer Category=Functional Equivalence Methods2026.05 | 0.147 | |
| Func. Neg.Model=Gem-Fl-Lt, Scorer Category=Functional Equivalence Methods2026.05 | 0.168 | |
| Func. Equiv. RateModel=Gem-Fl-Lt, Scorer Category=Functional Equivalence Methods2026.05 | 0.169 | |
| Func. SetsModel=GPT4o, Scorer Category=Functional Equivalence Methods2026.05 | 0.174 | |
| Min. Prob.Model=GPT4o-m, Scorer Category=Token-Probability Methods2026.05 | 0.175 | |
| Func. SetsModel=Gem-Fl-Lt, Scorer Category=Functional Equivalence Methods2026.05 | 0.176 | |
| Func. Neg.Model=GPT4o, Scorer Category=Functional Equivalence Methods2026.05 | 0.183 | |
| Min. Tok. Neg.Model=GPT4o, Scorer Category=Token-Probability Methods2026.05 | 0.184 | |
| Min. Prob.Model=Gem-Fl-Lt, Scorer Category=Token-Probability Methods2026.05 | 0.199 | |
| Min. Tok. Neg.Model=Gem-Fl-Lt, Scorer Category=Token-Probability Methods2026.05 | 0.2 | |
| Func. Equiv. RateModel=GPT4o, Scorer Category=Functional Equivalence Methods2026.05 | 0.2 | |
| Func. SetsModel=Gem-Pro, Scorer Category=Functional Equivalence Methods2026.05 | 0.207 | |
| Func. Neg.Model=Gem-Pro, Scorer Category=Functional Equivalence Methods2026.05 | 0.215 | |
| Min. Tok. Neg.Model=GPT4o-m, Scorer Category=Token-Probability Methods2026.05 | 0.225 | |
| Min. Tok. Neg.Model=Gem-Fl, Scorer Category=Token-Probability Methods2026.05 | 0.248 | |
| Func. SetsModel=Gem-Fl, Scorer Category=Functional Equivalence Methods2026.05 | 0.248 | |
| Func. Neg.Model=Gem-Fl, Scorer Category=Functional Equivalence Methods2026.05 | 0.261 | |
| Func. Equiv. RateModel=Gem-Pro, Scorer Category=Functional Equivalence Methods2026.05 | 0.263 | |
| P(True)Model=GPT4o, Scorer Category=Reflexive Methods2026.05 | 0.264 | |
| Min. Prob.Model=GPT4o, Scorer Category=Token-Probability Methods2026.05 | 0.266 | |
| Min. Tok. Neg.Model=Gem-Pro, Scorer Category=Token-Probability Methods2026.05 | 0.266 | |
| Min. Prob.Model=Gem-Fl, Scorer Category=Token-Probability Methods2026.05 | 0.302 | |
| Func. Equiv. RateModel=Gem-Fl, Scorer Category=Functional Equivalence Methods2026.05 | 0.302 | |
| Min. Prob.Model=Gem-Pro, Scorer Category=Token-Probability Methods2026.05 | 0.308 | |
| Verb. Conf.Model=Gem-Fl-Lt, Scorer Category=Reflexive Methods2026.05 | 0.368 | |
| P(True)Model=Gem-Fl-Lt, Scorer Category=Reflexive Methods2026.05 | 0.381 | |
| Seq. Prob.Model=GPT4o, Scorer Category=Token-Probability Methods2026.05 | 0.398 | |
| P(True)Model=Gem-Pro, Scorer Category=Reflexive Methods2026.05 | 0.422 | |
| Verb. Conf.Model=Gem-Fl, Scorer Category=Reflexive Methods2026.05 | 0.43 | |
| Verb. Conf.Model=GPT4o-m, Scorer Category=Reflexive Methods2026.05 | 0.431 | |
| Verb. Conf.Model=Gem-Pro, Scorer Category=Reflexive Methods2026.05 | 0.46 | |
| P(True)Model=GPT4o-m, Scorer Category=Reflexive Methods2026.05 | 0.477 | |
| P(True)Model=Gem-Fl, Scorer Category=Reflexive Methods2026.05 | 0.485 | |
| Cos. Sim.Model=Gem-Pro, Scorer Category=Similarity-Based Methods2026.05 | 0.503 | |
| Cos. Sim.Model=Gem-Fl, Scorer Category=Similarity-Based Methods2026.05 | 0.51 | |
| Prob. MarginModel=Gem-Pro, Scorer Category=Token-Probability Methods2026.05 | 0.518 | |
| Seq. Prob.Model=Gem-Pro, Scorer Category=Token-Probability Methods2026.05 | 0.519 | |
| Seq. Prob.Model=Gem-Fl, Scorer Category=Token-Probability Methods2026.05 | 0.52 | |
| Prob. MarginModel=Gem-Fl, Scorer Category=Token-Probability Methods2026.05 | 0.521 | |
| Avg. Tok. Neg.Model=Gem-Fl, Scorer Category=Token-Probability Methods2026.05 | 0.532 | |
| Avg. Tok. Neg.Model=Gem-Pro, Scorer Category=Token-Probability Methods2026.05 | 0.532 | |
| Verb. Conf.Model=GPT4o, Scorer Category=Reflexive Methods2026.05 | 0.548 | |
| Prob. MarginModel=GPT4o, Scorer Category=Token-Probability Methods2026.05 | 0.553 | |
| Avg. Tok. Neg.Model=GPT4o, Scorer Category=Token-Probability Methods2026.05 | 0.604 | |
| Cos. Sim.Model=GPT4o, Scorer Category=Similarity-Based Methods2026.05 | 0.62 | |
| Seq. Prob.Model=GPT4o-m, Scorer Category=Token-Probability Methods2026.05 | 0.627 | |
| Prob. MarginModel=GPT4o-m, Scorer Category=Token-Probability Methods2026.05 | 0.686 | |
| Seq. Prob.Model=Gem-Fl-Lt, Scorer Category=Token-Probability Methods2026.05 | 0.687 | |
| Prob. MarginModel=Gem-Fl-Lt, Scorer Category=Token-Probability Methods2026.05 | 0.692 | |
| Cos. Sim.Model=Gem-Fl-Lt, Scorer Category=Similarity-Based Methods2026.05 | 0.705 | |
| Avg. Tok. Neg.Model=Gem-Fl-Lt, Scorer Category=Token-Probability Methods2026.05 | 0.724 | |
| Avg. Tok. Neg.Model=GPT4o-m, Scorer Category=Token-Probability Methods2026.05 | 0.729 | |
| Cos. Sim.Model=GPT4o-m, Scorer Category=Similarity-Based Methods2026.05 | 0.734 |