Pragmatic Reasoning on LUDWIG
86.5AccuracyPRAGREST-GRPO
Evaluation results
| Method | Links | ||
|---|---|---|---|
| PRAGREST-GRPOBackbone=Qwen3-14B2026.06 | 86.5 | 0.3 | |
| PRAGREST-GRPOBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 86.5 | — | |
| Human score2026.06 | 86.2 | — | |
| PRAGREST-SFTBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 85.17 | — | |
| DEEP-LAYER-DPOBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 83.5 | — | |
| IMP-SFTBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 83.17 | — | |
| PRAGREST-GRPOBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 83 | — | |
| IMP-SFTBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 82.83 | — | |
| Non-counterfactual-GRPOBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 82.83 | — | |
| InstructBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 82.67 | — | |
| PRAGREST-SFTBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 82.17 | — | |
| Non-counterfactual-SFTBackbone=Qwen3-14B, Decoding=greedy decoding2026.06 | 81.17 | — | |
| InstructBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 80.33 | — | |
| Non-counterfactual-GRPOBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 80.17 | — | |
| DEEP-LAYER-DPOBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 80 | — | |
| Non-counterfactual-SFTBackbone=Qwen3-8B, Decoding=greedy decoding2026.06 | 79.5 | — |