Pragmatic Reasoning on PRAGMEGA
86.37AccuracyHuman score
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Human score2026.06 | 86.37 | — | |
| PRAGREST-GRPOBackbone=Qwen3-14B2026.06 | 85.8 | 0.57 | |
| PRAGREST-GRPOBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 85.8 | — | |
| PRAGREST-SFTBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 85.21 | — | |
| InstructBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 81.66 | — | |
| IMP-SFTBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 81.07 | — | |
| PRAGREST-GRPOBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 80.47 | — | |
| Non-counterfactual-SFTBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 80.47 | — | |
| Non-counterfactual-GRPOBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 78.7 | — | |
| PRAGREST-SFTBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 77.51 | — | |
| DEEP-LAYER-DPOBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 77.51 | — | |
| Non-counterfactual-SFTBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 73.96 | — | |
| InstructBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 73.37 | — | |
| Non-counterfactual-GRPOBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 72.19 | — | |
| IMP-SFTBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 69.82 | — | |
| DEEP-LAYER-DPOBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 67.46 | — |