Code Hallucination Detection on Odex
85OPMypy
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MypyQuery Model=claude-32026.04 | 85 | 15 | 17 | |
| MypyQuery Model=Granite2026.04 | 81 | 18 | 17 | |
| o3-miniQuery Model=gpt-42026.04 | 79 | 32 | 8 | |
| PyrightQuery Model=claude-32026.04 | 76 | 22 | 26 | |
| o3-miniQuery Model=Granite2026.04 | 74 | 34 | 34 | |
| PyrightQuery Model=Granite2026.04 | 72 | 34 | 36 | |
| o3-miniQuery Model=claude-32026.04 | 61 | 9 | 8 | |
| o3-miniQuery Model=gpt-3.52026.04 | 61 | 22 | 11 | |
| GrammarQuery Model=Granite2026.04 | 59 | 15 | 17 | |
| GrammarQuery Model=gpt-42026.04 | 52 | 11 | 12 | |
| MypyQuery Model=gpt-42026.04 | 43 | 81 | 70 | |
| PyrightQuery Model=gpt-42026.04 | 43 | 82 | 71 | |
| GrammarQuery Model=gpt-3.52026.04 | 43 | 8 | 10 | |
| MypyQuery Model=gpt-3.52026.04 | 39 | 66 | 60 | |
| PyrightQuery Model=gpt-3.52026.04 | 39 | 68 | 64 | |
| GrammarQuery Model=claude-32026.04 | 35 | 8 | 11 |