Compositional Reasoning on MAT-PREF (IID)
65.2Logical Question AccuracyQwen3-8B + SFT + GRPO
Evaluation results
| Method | Links | |
|---|---|---|
| Qwen3-8B + SFT + GRPOtraining_stage=SFT + GRPO2026.06 | 65.2 | |
| Qwen3-8B + SFT (SC@8)training_stage=SFT, evaluation_protocol=self-consistency SC@82026.06 | 50.6 | |
| Qwen3-8B + SFTtraining_stage=SFT2026.06 | 44.9 | |
| Qwen3-235B-Instructmode=zero-shot2026.06 | 43.8 | |
| Qwen2.5-72Bmode=zero-shot2026.06 | 37.1 | |
| DeepSeek-V3mode=zero-shot2026.06 | 36 | |
| Llama 3.3-70Bmode=zero-shot2026.06 | 33.7 | |
| Random baselineevaluation_protocol=majority voting across distractor permutations2026.06 | 21.9 |