Software Vulnerability Detection on OSV-Bench 1.0 (test)
97.18Pointer Errors (Pass@1)Claude Opus 4.6
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Claude Opus 4.6BODHI=true, Temperature=0.0, API calls=isolated per-task2026.04 | 97.18 | 95.54 | 94.59 | 98.15 | 96.3 | 97.96 | 96.73 | |
| Claude Sonnet 4BODHI=true, Temperature=0.0, API calls=isolated per-task2026.04 | 77.46 | 66.96 | 75.68 | 79.63 | 70.37 | 85.71 | 75.1 | |
| Claude Haiku 4.5BODHI=true, Temperature=0.0, API calls=isolated per-task2026.04 | 76.06 | 66.07 | 70.27 | 75.93 | 72.22 | 83.67 | 76.33 | |
| Devstral 2BODHI=true, Temperature=0.0, API calls=isolated per-task2026.04 | 74.65 | 72.32 | 75.68 | 85.19 | 77.78 | 85.71 | 80 | |
| Claude Opus 4.6BODHI=false, Temperature=0.0, API calls=isolated per-task2026.04 | 70.42 | 72.32 | 60.81 | 62.96 | 69.44 | 79.59 | 74.69 | |
| Nova PremierBODHI=true, Temperature=0.0, API calls=isolated per-task2026.04 | 56.34 | 53.57 | 52.7 | 37.04 | 57.41 | 63.27 | 58.78 | |
| Claude Sonnet 4BODHI=false, Temperature=0.0, API calls=isolated per-task2026.04 | 54.93 | 51.79 | 52.7 | 46.3 | 55.56 | 75.51 | 61.22 | |
| Qwen3BODHI=true, Temperature=0.0, API calls=isolated per-task2026.04 | 54.93 | 37.5 | 44.59 | 38.89 | 47.22 | 55.1 | 49.8 | |
| Llama 4 MaverickBODHI=true, Temperature=0.0, API calls=isolated per-task2026.04 | 52.11 | 48.21 | 39.19 | 46.3 | 57.41 | 59.18 | 55.1 | |
| Doubao-1.5-proSource=Published [1], BODHI=false2026.04 | 50.7 | 48.21 | 45.95 | 40.74 | 52.78 | 63.27 | 55.1 | |
| DeepSeek-V3.2BODHI=true, Temperature=0.0, API calls=isolated per-task2026.04 | 47.89 | 53.57 | 51.35 | 59.26 | 52.78 | 63.27 | 56.73 | |
| Devstral 2BODHI=false, Temperature=0.0, API calls=isolated per-task2026.04 | 46.48 | 41.96 | 47.3 | 42.59 | 48.15 | 51.02 | 48.16 | |
| Qwen3BODHI=false, Temperature=0.0, API calls=isolated per-task2026.04 | 43.66 | 29.46 | 32.43 | 20.37 | 33.33 | 44.9 | 38.78 | |
| Claude Sonnet 3.5Source=Published [1], BODHI=false2026.04 | 39.44 | 41.96 | 39.19 | 48.15 | 39.81 | 46.94 | 44.9 | |
| Claude Haiku 4.5BODHI=false, Temperature=0.0, API calls=isolated per-task2026.04 | 39.44 | 41.96 | 41.89 | 42.59 | 43.52 | 59.18 | 50.2 | |
| Nova PremierBODHI=false, Temperature=0.0, API calls=isolated per-task2026.04 | 39.44 | 35.71 | 37.84 | 27.78 | 40.74 | 48.98 | 43.67 | |
| DeepSeek-ChatSource=Published [1], BODHI=false2026.04 | 38.02 | 39.29 | 36.49 | 44.44 | 43.52 | 51.02 | 46.53 | |
| DeepSeek-R1†BODHI=true, Temperature=not configurable, API calls=isolated per-task, Type=Reasoning model2026.04 | 35.21 | 33.93 | 27.03 | 44.44 | 37.04 | 53.06 | 40.41 | |
| Llama 4 MaverickBODHI=false, Temperature=0.0, API calls=isolated per-task2026.04 | 33.8 | 33.04 | 25.68 | 35.19 | 40.74 | 42.86 | 38.78 | |
| DeepSeek-V3.2BODHI=false, Temperature=0.0, API calls=isolated per-task2026.04 | 30.99 | 36.61 | 31.08 | 29.63 | 34.26 | 38.78 | 38.78 | |
| DeepSeek-R1†BODHI=false, Temperature=not configurable, API calls=isolated per-task, Type=Reasoning model2026.04 | 21.13 | 22.32 | 18.92 | 29.63 | 28.7 | 24.49 | 26.53 |