Counterfactual Reasoning on CounterBench
80.8Basic ScoreDeepseek-V3
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Deepseek-V3Reasoning Strategy=CausalCoT2025.02 | 80.8 | 70.4 | 76.4 | 77.6 | 63.5 | 74.2 | |
| GPT-4oReasoning Strategy=CausalCoT2025.02 | 80.4 | 72.4 | 80.8 | 81.6 | 60.5 | 75.8 | |
| Gemini-1.5-flashReasoning Strategy=Standard2025.02 | 75.2 | 65.6 | 67.2 | 76 | 53 | 68 | |
| Gemini-1.5-flashReasoning Strategy=CausalCoT2025.02 | 72.4 | 70.8 | 72.4 | 78.4 | 58.5 | 71 | |
| Gemini-1.5-flash-8bReasoning Strategy=CausalCoT2025.02 | 66.8 | 67.2 | 65.2 | 65.2 | 58.5 | 64.8 | |
| Claude-3.5 (Haiku)Reasoning Strategy=CausalCoT2025.02 | 60.4 | 65.6 | 67.2 | 66 | 61 | 64.2 | |
| Claude-3 (Sonnet)Reasoning Strategy=CausalCoT2025.02 | 59.2 | 52 | 64.4 | 60 | 65.5 | 59 | |
| GPT-4o miniReasoning Strategy=CausalCoT2025.02 | 57.2 | 66.4 | 60 | 63.2 | 50 | 59.8 | |
| GPT-3 (Davinci-002)Reasoning Strategy=Standard2025.02 | 56.8 | 50.2 | 48.8 | 51.6 | 52.5 | 51.9 | |
| GPT-3 (Davinci-002)Reasoning Strategy=CausalCoT2025.02 | 51.2 | 41.9 | 51.2 | 51.6 | 50.5 | 49.3 | |
| GPT-4oReasoning Strategy=Standard2025.02 | 50.4 | 54.4 | 50.4 | 54.8 | 54 | 52.8 | |
| Claude-3 (Sonnet)Reasoning Strategy=Standard2025.02 | 50.4 | 48.8 | 50 | 50.8 | 59.5 | 51.6 | |
| Deepseek-V3Reasoning Strategy=Standard2025.02 | 50.4 | 50.4 | 50 | 50 | 60.5 | 51.9 | |
| GPT-3 (Babbage-002)Reasoning Strategy=Standard2025.02 | 50 | 50 | 50 | 50 | 47.5 | 49.6 | |
| GPT-4o miniReasoning Strategy=Standard2025.02 | 50 | 50 | 50 | 50 | 52.5 | 50.4 | |
| Gemini-1.5-flash-8bReasoning Strategy=Standard2025.02 | 50 | 50 | 50 | 50 | 52.5 | 50.4 | |
| GPT-3.5Reasoning Strategy=Standard2025.02 | 49.6 | 51.2 | 50.4 | 50 | 52 | 50.6 | |
| GPT-3.5Reasoning Strategy=CausalCoT2025.02 | 43.6 | 50.4 | 53.6 | 50 | 47.5 | 49.1 | |
| Claude-3.5 (Haiku)Reasoning Strategy=Standard2025.02 | 28.4 | 24 | 43.6 | 54 | 51 | 39.8 | |
| GPT-3 (Babbage-002)Reasoning Strategy=CausalCoT2025.02 | 3.6 | 7.6 | 1.2 | 19.6 | 18.5 | 9.8 |