Rule Discovery on Blicket Task
86Task Success RateQwen3-32B
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Qwen3-32BThinking Mode=Thinking, Prompt Strategy=TiO2026.04 | 86 | 12.98 | 1.31 | |
| LLaMA-3.3-70B-InsThinking Mode=Non-Thinking, Prompt Strategy=TiO2026.04 | 82 | 13.5 | 0.98 | |
| QwQ-32BThinking Mode=Thinking, Prompt Strategy=TiO2026.04 | 82 | 14.68 | 1.3 | |
| QwQ-32BThinking Mode=Thinking, Prompt Strategy=Baseline2026.04 | 80 | 12.27 | 0.89 | |
| LLaMA-3.3-70B-InsThinking Mode=Non-Thinking, Prompt Strategy=Baseline2026.04 | 78 | 12.91 | 0.82 | |
| Qwen3-32BThinking Mode=Non-Thinking, Prompt Strategy=Baseline2026.04 | 64 | 14 | 1 | |
| Qwen3-14BThinking Mode=Thinking, Prompt Strategy=TiO2026.04 | 61 | 12.33 | 0.48 | |
| Qwen3-32BThinking Mode=Thinking, Prompt Strategy=Baseline2026.04 | 57 | 12.18 | 0.54 | |
| Qwen3-32BThinking Mode=Non-Thinking, Prompt Strategy=TiO2026.04 | 56 | 13.89 | 1.33 | |
| Qwen3-14BThinking Mode=Thinking, Prompt Strategy=Baseline2026.04 | 50 | 9.48 | 0.34 | |
| DeepSeek-R1-LLaMA-70BThinking Mode=Thinking, Prompt Strategy=TiO2026.04 | 43 | 17.41 | 0.57 | |
| DeepSeek-R1-LLaMA-70BThinking Mode=Thinking, Prompt Strategy=Baseline2026.04 | 41 | 12.93 | 0.43 | |
| Qwen3-8BThinking Mode=Thinking, Prompt Strategy=TiO2026.04 | 40 | 10.73 | 0.57 | |
| Qwen3-8BThinking Mode=Thinking, Prompt Strategy=Baseline2026.04 | 32 | 10.26 | 0.44 | |
| Qwen3-14BThinking Mode=Non-Thinking, Prompt Strategy=Baseline2026.04 | 25 | 10.38 | 0.55 | |
| Qwen3-14BThinking Mode=Non-Thinking, Prompt Strategy=TiO2026.04 | 20 | 10.73 | 0.74 | |
| Qwen3-8BThinking Mode=Non-Thinking, Prompt Strategy=TiO2026.04 | 19 | 12.14 | 0.25 | |
| Qwen3-8BThinking Mode=Non-Thinking, Prompt Strategy=Baseline2026.04 | 11 | 10.24 | 0.36 |