Shortcut detection on BenchMarker Shortcuts
81.6AccuracyGPT-5
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GPT-52026.02 | 81.6 | 75.3 | 60.6 | |
| Claude 4.5Model Variant=Sonnet2026.02 | 80.8 | 74.8 | 59.3 | |
| Claude 4.5Model Variant=Haiku2026.02 | 78.2 | 73.1 | 55.1 | |
| GPT-5Model Variant=Mini2026.02 | 76.8 | 72.3 | 52.8 | |
| Gemini 2.5Model Variant=Lite2026.02 | 75.7 | 74 | 53.1 | |
| Command R2026.02 | 74.2 | 72.2 | 50.1 | |
| Gemma-3Parameters=27B2026.02 | 74.2 | 72.4 | 50.3 | |
| Qwen-3Parameters=32B2026.02 | 73.4 | 70.7 | 48.1 | |
| Command R+2026.02 | 72.3 | 69.6 | 46 | |
| Qwen-3Parameters=0.6B2026.02 | 70.5 | 63.6 | 39.2 | |
| Gemini 2.5Model Variant=Pro2026.02 | 70.1 | 68.7 | 43 | |
| Gemini 2.5Model Variant=Flash2026.02 | 68.6 | 67.7 | 40.6 | |
| GPT-5Model Variant=Nano2026.02 | 67.9 | 65.1 | 37.6 | |
| Always Not Flawed2026.02 | 63.5 | 0 | 0 | |
| Qwen-3Parameters=1.7B2026.02 | 62.4 | 63.6 | 31 | |
| Qwen-3Parameters=14B2026.02 | 61.3 | 63.7 | 30.1 | |
| Gemma-3Parameters=12B2026.02 | 61.3 | 64.2 | 30.6 | |
| LLaMA-3.1Parameters=70B2026.02 | 60.5 | 61.9 | 27.8 | |
| LLaMA-3.2Parameters=3B2026.02 | 57.9 | 62.3 | 25.8 | |
| Qwen-3Parameters=8B2026.02 | 57.2 | 61.3 | 24.3 | |
| LLaMA-3.2Parameters=1B2026.02 | 56.8 | 29.9 | 0.3 | |
| Gemma-3Parameters=4B2026.02 | 50.2 | 58.5 | 15.6 | |
| Random (50/50)2026.02 | 50 | 42.2 | 0 | |
| LLaMA-3.1Parameters=8B2026.02 | 48.7 | 58.3 | 14.2 | |
| Qwen-3Parameters=4B2026.02 | 42.1 | 54.5 | 5 | |
| Always Flawed2026.02 | 36.5 | 53.5 | 0 |