Logical Fallacy Detection on LOGIC-Climate 10% representative subset (107 instances)
302CountGemini 2.5 pro
Evaluation Results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Gemini 2.5 proMode=Thinking (T), Executor=Claude-sonnet-4.5, Max repair iterations=42026.06 | 302 | 91.72 | 17.88 | 73.84 | 0.99 | 7.62 | 3.31 | 0.66 | 3.64 | |
| GPT-5Mode=Thinking (T), Executor=Claude-sonnet-4.5, Max repair iterations=42026.06 | 290 | 97.59 | 17.93 | 79.66 | 0.34 | 2.07 | 0.69 | 1.38 | 0 | |
| Grok 4 fastMode=Non-Thinking (NT), Executor=Claude-sonnet-4.5, Max repair iterations=42026.06 | 279 | 96.41 | 17.92 | 78.49 | 1.08 | 2.87 | 1.79 | 0.36 | 0.72 | |
| Kimi-k2Mode=Thinking (T), Executor=Claude-sonnet-4.5, Max repair iterations=42026.06 | 276 | 94.2 | 18.48 | 75.72 | 1.45 | 5.43 | 0.72 | 3.62 | 1.09 | |
| Qwen3 235BMode=Non-Thinking (NT), Executor=Claude-sonnet-4.5, Max repair iterations=42026.06 | 259 | 94.59 | 22.39 | 72.2 | 1.16 | 4.25 | 1.93 | 1.16 | 1.16 | |
| Gemini 2.5 proMode=Non-Thinking (NT), Executor=Claude-sonnet-4.5, Max repair iterations=42026.06 | 248 | 91.13 | 19.76 | 71.37 | 2.02 | 6.85 | 3.23 | 0.4 | 3.23 | |
| Qwen3 235BMode=Thinking (T), Executor=Claude-sonnet-4.5, Max repair iterations=42026.06 | 237 | 92.4 | 20.25 | 72.15 | 0 | 7.59 | 2.11 | 2.95 | 2.53 | |
| GPT-4oMode=Non-Thinking (NT), Executor=Claude-sonnet-4.5, Max repair iterations=42026.06 | 236 | 96.61 | 22.88 | 73.73 | 0 | 3.39 | 0.85 | 2.54 | 0 | |
| GPT-oss-120bMode=Thinking (T), Executor=Claude-sonnet-4.5, Max repair iterations=42026.06 | 234 | 97.01 | 20.51 | 76.5 | 0 | 2.99 | 0 | 2.99 | 0 | |
| Gemini 2.5 flashMode=Thinking (T), Executor=Claude-sonnet-4.5, Max repair iterations=42026.06 | 229 | 89.52 | 21.4 | 68.12 | 2.62 | 7.86 | 6.11 | 0.44 | 1.31 | |
| Claude-sonnet-4.5Mode=Thinking (T), Executor=Claude-sonnet-4.5, Max repair iterations=42026.06 | 227 | 93.84 | 22.91 | 70.93 | 0.88 | 5.29 | 2.2 | 2.64 | 0.44 | |
| Grok 4 fastMode=Thinking (T), Executor=Claude-sonnet-4.5, Max repair iterations=42026.06 | 215 | 90.7 | 20.93 | 69.77 | 0.47 | 8.84 | 2.33 | 5.12 | 1.4 | |
| Deepseek-r1Mode=Thinking (T), Executor=Claude-sonnet-4.5, Max repair iterations=42026.06 | 201 | 92.04 | 22.39 | 69.65 | 0.5 | 7.96 | 0 | 5.97 | 1.99 | |
| Gemini 2.5 flashMode=Non-Thinking (NT), Executor=Claude-sonnet-4.5, Max repair iterations=42026.06 | 195 | 90.77 | 22.05 | 68.72 | 1.03 | 8.21 | 3.08 | 4.1 | 1.03 | |
| Deepseek-chat-v3.1Mode=Thinking (T), Executor=Claude-sonnet-4.5, Max repair iterations=42026.06 | 193 | 90.16 | 20.21 | 69.95 | 1.04 | 9.33 | 1.04 | 5.7 | 2.59 |