Code Hallucination Detection on BigCode
0.77OPo3-mini
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| o3-miniQuery Model=Granite2026.04 | 0.77 | 0.48 | 0.5 | |
| o3-miniQuery Model=claude-32026.04 | 0.64 | 0.12 | 0.15 | |
| PyrightQuery Model=Granite2026.04 | 0.62 | 0.3 | 0.36 | |
| o3-miniQuery Model=gpt-42026.04 | 0.57 | 0.11 | 0.14 | |
| o3-miniQuery Model=gpt-3.52026.04 | 0.56 | 0.16 | 0.14 | |
| MypyQuery Model=Granite2026.04 | 0.47 | 0.09 | 0.1 | |
| GrammarQuery Model=Granite2026.04 | 0.45 | 0.09 | 0.06 | |
| PyrightQuery Model=gpt-42026.04 | 0.34 | 0.2 | 0.24 | |
| PyrightQuery Model=claude-32026.04 | 0.32 | 0.2 | 0.24 | |
| PyrightQuery Model=gpt-3.52026.04 | 0.28 | 0.16 | 0.14 | |
| MypyQuery Model=gpt-42026.04 | 0.24 | 0.09 | 0.08 | |
| GrammarQuery Model=gpt-3.52026.04 | 0.24 | 0.12 | 0.11 | |
| GrammarQuery Model=gpt-42026.04 | 0.23 | 0.06 | 0.06 | |
| MypyQuery Model=gpt-3.52026.04 | 0.22 | 0.07 | 0.03 | |
| GrammarQuery Model=claude-32026.04 | 0.19 | 0.04 | 0.06 | |
| MypyQuery Model=claude-32026.04 | 0.16 | 0.05 | 0.03 |