Reasoning on BIG-Bench Hard (5 subtasks full)
86AccuracyThinking
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ThinkingBudget=2048, Backbone=Qwen3-8B, Thinking Mode=True2026.05 | 86 | -11 | |
| NothinkBudget=1024, Backbone=Qwen3-8B, Thinking Mode=False2026.05 | 75.1 | 1.4 | |
| NothinkBudget=2048, Backbone=Qwen3-8B, Thinking Mode=False2026.05 | 75.1 | -11 | |
| ThinkingBudget=1024, Backbone=Qwen3-8B, Thinking Mode=True2026.05 | 73.6 | 1.4 | |
| NothinkBudget=512, Backbone=Qwen3-8B, Thinking Mode=False2026.05 | 66.4 | 20.6 | |
| NothinkBudget=256, Backbone=Qwen3-8B, Thinking Mode=False2026.05 | 49.9 | 33.3 | |
| ThinkingBudget=512, Backbone=Qwen3-8B, Thinking Mode=True2026.05 | 45.8 | 20.6 | |
| ThinkingBudget=256, Backbone=Qwen3-8B, Thinking Mode=True2026.05 | 16.6 | 33.3 |