Compositional Reasoning on MAT-PREF (OOD-host)
71.6Logical Question AccuracyQwen3-8B + SFT + GRPO
Evaluation results
| Method | Links | |
|---|---|---|
| Qwen3-8B + SFT + GRPOtraining_stage=SFT + GRPO2026.06 | 71.6 | |
| Qwen3-8B + SFT (SC@8)training_stage=SFT, evaluation_protocol=self-consistency SC@82026.06 | 50.4 | |
| Qwen3-8B + SFTtraining_stage=SFT2026.06 | 47.9 | |
| Qwen3-235B-Instructmode=zero-shot2026.06 | 45.4 | |
| DeepSeek-V3mode=zero-shot2026.06 | 36.5 | |
| Llama 3.3-70Bmode=zero-shot2026.06 | 35.1 | |
| Qwen2.5-72Bmode=zero-shot2026.06 | 33.3 | |
| Random baselineevaluation_protocol=majority voting across distractor permutations2026.06 | 25 |