Code Hallucination Detection on DS-1000
0.95OPPyright
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| PyrightQuery Model=Granite2026.04 | 0.95 | 0.55 | 0.52 | |
| o3-miniQuery Model=Granite2026.04 | 0.79 | 0.51 | 0.68 | |
| MypyQuery Model=Granite2026.04 | 0.63 | 0.26 | 0.21 | |
| o3-miniQuery Model=claude-32026.04 | 0.63 | 0.43 | 0.29 | |
| PyrightQuery Model=claude-32026.04 | 0.61 | 0.7 | 0.85 | |
| o3-miniQuery Model=gpt-3.52026.04 | 0.61 | 0.45 | 0.29 | |
| PyrightQuery Model=gpt-3.52026.04 | 0.57 | 0.6 | 0.75 | |
| GrammarQuery Model=Granite2026.04 | 0.57 | 0.1 | 0.17 | |
| o3-miniQuery Model=gpt-42026.04 | 0.55 | 0.43 | 0.26 | |
| GrammarQuery Model=gpt-42026.04 | 0.53 | 0.15 | 0.2 | |
| PyrightQuery Model=gpt-42026.04 | 0.5 | 0.62 | 0.84 | |
| GrammarQuery Model=gpt-3.52026.04 | 0.45 | 0.11 | 0.15 | |
| GrammarQuery Model=claude-32026.04 | 0.36 | 0.14 | 0.16 | |
| MypyQuery Model=gpt-3.52026.04 | 0.26 | 0.34 | 0.23 | |
| MypyQuery Model=claude-32026.04 | 0.22 | 0.27 | 0.22 | |
| MypyQuery Model=gpt-42026.04 | 0.16 | 0.23 | 0.26 |