Pragmatic Reasoning on METOQA
80.31AccuracyPRAGREST-GRPO
Evaluation results
| Method | Links | ||
|---|---|---|---|
| PRAGREST-GRPOBackbone=Qwen3-14B2026.06 | 80.31 | 0.31 | |
| PRAGREST-GRPOBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 80.31 | — | |
| Human score2026.06 | 80 | — | |
| PRAGREST-GRPOBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 79.87 | — | |
| PRAGREST-SFTBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 79.65 | — | |
| PRAGREST-SFTBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 78.56 | — | |
| Non-counterfactual-SFTBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 73.74 | — | |
| InstructBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 73.52 | — | |
| IMP-SFTBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 73.09 | — | |
| DEEP-LAYER-DPOBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 72.87 | — | |
| DEEP-LAYER-DPOBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 72.65 | — | |
| Non-counterfactual-GRPOBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 72.43 | — | |
| IMP-SFTBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 72.21 | — | |
| Non-counterfactual-SFTBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 71.99 | — | |
| InstructBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 71.77 | — | |
| Non-counterfactual-GRPOBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 71.55 | — |