Logic Reasoning on ZebraLogic (Accuracy and Latency)
96AccuracyBaseline (Thinking)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Baseline (Thinking)Model=Qwen3-30B-A3B (2507)2025.12 | 96 | 1,244.08 | 1,244.08 | |
| Qwen3-Base (DICE-PC)Size=3×4B2026.06 | 93.1 | — | — | |
| AsyncReasoningModel=Qwen3-30B-A3B (2507)2025.12 | 93 | 3.72 | 3.72 | |
| OpenAI o3Size=–2026.06 | 92.1 | — | — | |
| Qwen3-235B-A22BSize=235B2026.06 | 91.7 | — | — | |
| Qwen3-Base (DICE-FT)Size=3×4B2026.06 | 91.1 | — | — | |
| Qwen3-Inst (SC@64)Size=14B2026.06 | 90.3 | — | — | |
| Qwen3-A3BSize=30B2026.06 | 89.5 | — | — | |
| OpenAI o3-miniSize=–2026.06 | 88.9 | — | — | |
| Qwen3 (thinking)Size=32B2026.06 | 88.8 | — | — | |
| Baseline (Thinking)Model=Qwen3-235B-A22B (2507)2025.12 | 88 | 2,111.85 | 2,111.67 | |
| Qwen3-Inst (Few-shot CoT)Size=4B2026.06 | 87.7 | — | — | |
| Qwen3-Base Debate+Judge (FT)Size=3×8B2026.06 | 86.3 | — | — | |
| Qwen3-Inst Debate+Judge (prompt)Size=3×4B2026.06 | 85.9 | — | — | |
| Qwen3-Base Role-play (FT)Size=3×8B2026.06 | 85.1 | — | — | |
| AsyncReasoningModel=Qwen3-235B-A22B (2507)2025.12 | 85 | 57.85 | 8.37 | |
| Qwen3-Inst Role-play (prompt)Size=3×4B2026.06 | 84.6 | — | — | |
| Qwen3-Inst (ReAct)Size=4B2026.06 | 84.2 | — | — | |
| Qwen3-Base Debate (FT)Size=3×8B2026.06 | 84.1 | — | — | |
| Qwen3-Base Debate+Judge (FT)Size=3×4B2026.06 | 83.8 | — | — | |
| Qwen3-Inst Debate (prompt)Size=3×4B2026.06 | 83.7 | — | — | |
| Qwen3-Base Role-play (FT)Size=3×4B2026.06 | 82.6 | — | — | |
| Qwen3-Inst (SC@64)Size=4B2026.06 | 82.1 | — | — | |
| Qwen3-Inst (ToT)Size=4B2026.06 | 81.9 | — | — | |
| Qwen3-Base Debate (FT)Size=3×4B2026.06 | 81.7 | — | — | |
| Baseline (Thinking)Model=gpt-oss-120b2025.12 | 81 | 268.83 | 268.83 | |
| Llama 3.1-Base (DICE-PC)Size=3×8B2026.06 | 80.7 | — | — | |
| QwQSize=32B2026.06 | 76.8 | — | — | |
| AsyncReasoningModel=gpt-oss-120b2025.12 | 75 | 131.57 | 7.53 | |
| DeepSeek-R1 DistillSize=70B2026.06 | 71.3 | — | — | |
| Baseline (Thinking)Model=Qwen3-32B2025.12 | 71 | 297.61 | 297.61 | |
| Baseline (Thinking)Model=gpt-oss-20b2025.12 | 70 | 372.41 | 372.41 | |
| AsyncReasoningModel=Qwen3-32B2025.12 | 68 | 15.83 | 4.37 | |
| Qwen3-Inst (Zero-shot CoT)Size=4B2026.06 | 68 | — | — | |
| Baseline (Interactive)Model=Qwen3-235B-A22B (2507), Protocol=No thinking2025.12 | 67 | 4.68 | 4.53 | |
| AsyncReasoningModel=gpt-oss-20b2025.12 | 61 | 163.34 | 4.37 | |
| Baseline (Interactive)Model=gpt-oss-120b, Protocol=Low budget2025.12 | 48 | 248.13 | 248.13 | |
| Qwen3-Base-DAPOSize=14B2026.06 | 40.2 | — | — | |
| Qwen3-Base-GRPOSize=14B2026.06 | 38.7 | — | — | |
| Baseline (Interactive)Model=Qwen3-32B, Protocol=No thinking2025.12 | 37 | 4.27 | 4.27 | |
| Qwen3-Base-DAPOSize=8B2026.06 | 36.7 | — | — | |
| Qwen3-Base-GRPOSize=8B2026.06 | 35.4 | — | — | |
| Qwen3-Base-PPOSize=8B2026.06 | 34 | — | — | |
| Qwen3-BaseSize=14B2026.06 | 33.1 | — | — | |
| Qwen3-BaseSize=32B2026.06 | 29.2 | — | — | |
| Baseline (Interactive)Model=Qwen3-30B-A3B (2507), Protocol=No thinking2025.12 | 27 | 4.16 | 4.16 | |
| Qwen2.5-BaseSize=72B2026.06 | 26.6 | — | — | |
| GPT-4o-miniSize=–2026.06 | 20.1 | — | — | |
| Baseline (Interactive)Model=gpt-oss-20b, Protocol=Low budget2025.12 | 1 | 149.44 | 149.44 |