Compositional Reasoning on MAT-PREF OOD-property
60Logical Question AccuracyQwen3-8B + SFT + GRPO
Evaluation results
| Method | Links | |
|---|---|---|
| Qwen3-8B + SFT + GRPOtraining_stage=SFT + GRPO2026.06 | 60 | |
| Qwen3-235B-Instructmode=zero-shot2026.06 | 53.3 | |
| Qwen3-8B + SFT (SC@8)training_stage=SFT, evaluation_protocol=self-consistency SC@82026.06 | 51.8 | |
| DeepSeek-V3mode=zero-shot2026.06 | 50.6 | |
| Qwen3-8B + SFTtraining_stage=SFT2026.06 | 50.2 | |
| Llama 3.3-70Bmode=zero-shot2026.06 | 46.3 | |
| Qwen2.5-72Bmode=zero-shot2026.06 | 39.6 | |
| Random baselineevaluation_protocol=majority voting across distractor permutations2026.06 | 23.4 |