Rule discovery on Wason's rule discovery game
78Task Success RateThink-in-Opposites (TiO)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Think-in-Opposites (TiO)Model=Gemini-2.5-Pro, Thinking Mode=Thinking2026.04 | 78 | 8.52 | — | 75 | |
| Dual-GoalModel=Gemini-2.5-Pro, Thinking Mode=Thinking2026.04 | 78 | 10.1 | — | 77 | |
| BaselineModel=Gemini-2.5-Pro, Thinking Mode=Thinking2026.04 | 73 | 11.14 | — | 66 | |
| Think-in-Opposites (TiO)Model=Qwen3-32B, Thinking Mode=Thinking2026.04 | 61 | 6.38 | 1,450 | 54 | |
| Dual-GoalModel=Qwen3-32B, Thinking Mode=Thinking2026.04 | 60 | 10.29 | 1,472.3 | 77 | |
| Think-in-Opposites (TiO)Model=QwQ-32B, Thinking Mode=Thinking2026.04 | 59 | 10.28 | 1,464.5 | 37 | |
| Dual-GoalModel=Qwen3-14B, Thinking Mode=Thinking2026.04 | 58 | 7.66 | 1,191.7 | 40 | |
| Dual-GoalModel=QwQ-32B, Thinking Mode=Thinking2026.04 | 58 | 10.76 | 2,277.2 | 81 | |
| Dual-GoalModel=DeepSeek-R1-LLaMA-70B, Thinking Mode=Thinking2026.04 | 50 | 8.39 | 1,211.2 | 61 | |
| BaselineModel=QwQ-32B, Thinking Mode=Thinking2026.04 | 50 | 11.75 | 1,757.5 | 36 | |
| Think-in-Opposites (TiO)Model=Qwen3-14B, Thinking Mode=Thinking2026.04 | 48 | 7.08 | 1,332.4 | 26 | |
| Think-in-Opposites (TiO)Model=DeepSeek-R1-LLaMA-70B, Thinking Mode=Thinking2026.04 | 46 | 10.07 | 1,072.8 | 48 | |
| BaselineModel=Qwen3-32B, Thinking Mode=Thinking2026.04 | 41 | 10.83 | 1,029.8 | 14 | |
| Dual-GoalModel=Qwen3-8B, Thinking Mode=Thinking2026.04 | 40 | 7.37 | 2,142.7 | 55 | |
| Think-in-Opposites (TiO)Model=LLaMA-3.3-70B-Ins, Thinking Mode=Non-Thinking2026.04 | 38 | 10.03 | 14.7 | 50 | |
| Think-in-Opposites (TiO)Model=Qwen3-8B, Thinking Mode=Thinking2026.04 | 36 | 9.01 | 1,716.1 | 35 | |
| Think-in-Opposites (TiO)Model=Qwen3-32B, Thinking Mode=Non-Thinking2026.04 | 36 | 3.76 | 6.1 | 63 | |
| BaselineModel=DeepSeek-R1-LLaMA-70B, Thinking Mode=Thinking2026.04 | 35 | 8.1 | 957.6 | 32 | |
| BaselineModel=Qwen3-14B, Thinking Mode=Thinking2026.04 | 34 | 7.39 | 1,179 | 19 | |
| BaselineModel=Qwen3-32B, Thinking Mode=Non-Thinking2026.04 | 33 | 4.93 | 6.6 | 50 | |
| Think-in-Opposites (TiO)Model=Qwen3-14B, Thinking Mode=Non-Thinking2026.04 | 29 | 4.12 | 7 | 90 | |
| BaselineModel=Qwen3-14B, Thinking Mode=Non-Thinking2026.04 | 23 | 3 | 7.6 | 72 | |
| BaselineModel=Qwen3-8B, Thinking Mode=Thinking2026.04 | 21 | 10.88 | 1,088.2 | 15 | |
| BaselineModel=LLaMA-3.3-70B-Ins, Thinking Mode=Non-Thinking2026.04 | 21 | 9.37 | 14.1 | 37 | |
| Dual-GoalModel=LLaMA-3.3-70B-Ins, Thinking Mode=Non-Thinking2026.04 | 20 | 12.87 | 23.7 | 53 | |
| Think-in-Opposites (TiO)Model=GPT-4o, Thinking Mode=Non-Thinking2026.04 | 16 | 2.5 | 10.7 | 17 | |
| Dual-GoalModel=GPT-4o, Thinking Mode=Non-Thinking2026.04 | 14 | 6.75 | 17.1 | 34 | |
| Dual-GoalModel=Qwen3-14B, Thinking Mode=Non-Thinking2026.04 | 13 | 0.5 | 15 | 41 | |
| BaselineModel=GPT-4o, Thinking Mode=Non-Thinking2026.04 | 11 | 0 | 10.4 | 25 | |
| Dual-GoalModel=Qwen3-32B, Thinking Mode=Non-Thinking2026.04 | 10 | 0 | 15.4 | 46 | |
| Dual-GoalModel=Qwen3-8B, Thinking Mode=Non-Thinking2026.04 | 8 | 0 | 13.4 | 57 | |
| BaselineModel=Qwen3-8B, Thinking Mode=Non-Thinking2026.04 | 6 | 5 | 8.4 | 35 | |
| Think-in-Opposites (TiO)Model=Qwen3-8B, Thinking Mode=Non-Thinking2026.04 | 6 | 15.25 | 8.4 | 30 |