Pragmatic Reasoning on PRAGMEGA (test)
94AccuracyGricean Prompting
Evaluation Results
| Method | Links | |
|---|---|---|
| Gricean PromptingBackbone=gpt-4o, Prompt Setting=Proposed, Prompting Strategy=grice2025.10 | 94 | |
| Relevance Theory PromptingBackbone=gpt-4o, Prompt Setting=Proposed, Prompting Strategy=relevance2025.10 | 93.5 | |
| Gricean PromptingBackbone=gpt-4o, Prompt Setting=Short, Prompting Strategy=grice2025.10 | 90.2 | |
| Relevance Theory PromptingBackbone=gpt-4o, Prompt Setting=Short, Prompting Strategy=relevance2025.10 | 89.2 | |
| Chain-of-ThoughtBackbone=gpt-4o, Prompt Setting=Baseline, Prompting Strategy=cot2025.10 | 87.7 | |
| Gricean PromptingBackbone=phi-4, Prompt Setting=Proposed, Prompting Strategy=grice2025.10 | 87.7 | |
| Relevance Theory PromptingBackbone=phi-4, Prompt Setting=Proposed, Prompting Strategy=relevance2025.10 | 86.7 | |
| Simple PromptingBackbone=gpt-4o, Prompt Setting=Baseline, Prompting Strategy=simple2025.10 | 84.2 | |
| Gricean PromptingBackbone=Qwen2.5-14B-Instruct, Prompt Setting=Proposed, Prompting Strategy=grice2025.10 | 79.6 | |
| Gricean PromptingBackbone=phi-4, Prompt Setting=Short, Prompting Strategy=grice2025.10 | 79.6 | |
| Relevance Theory PromptingBackbone=phi-4, Prompt Setting=Short, Prompting Strategy=relevance2025.10 | 79.2 | |
| Simple PromptingBackbone=phi-4, Prompt Setting=Baseline, Prompting Strategy=simple2025.10 | 78.1 | |
| Relevance Theory PromptingBackbone=gpt-4o-mini, Prompt Setting=Proposed, Prompting Strategy=relevance2025.10 | 77.9 | |
| Chain-of-ThoughtBackbone=phi-4, Prompt Setting=Baseline, Prompting Strategy=cot2025.10 | 77.5 | |
| Gricean PromptingBackbone=gpt-4o-mini, Prompt Setting=Proposed, Prompting Strategy=grice2025.10 | 77.1 | |
| Relevance Theory PromptingBackbone=Qwen2.5-14B-Instruct, Prompt Setting=Proposed, Prompting Strategy=relevance2025.10 | 75 | |
| Relevance Theory PromptingBackbone=gpt-4o-mini, Prompt Setting=Short, Prompting Strategy=relevance2025.10 | 73.8 | |
| Gricean PromptingBackbone=gpt-4o-mini, Prompt Setting=Short, Prompting Strategy=grice2025.10 | 72.3 | |
| Gricean PromptingBackbone=Qwen2.5-14B-Instruct, Prompt Setting=Short, Prompting Strategy=grice2025.10 | 72.3 | |
| Relevance Theory PromptingBackbone=Qwen2.5-14B-Instruct, Prompt Setting=Short, Prompting Strategy=relevance2025.10 | 72.1 | |
| Simple PromptingBackbone=Qwen2.5-14B-Instruct, Prompt Setting=Baseline, Prompting Strategy=simple2025.10 | 71.7 | |
| Simple PromptingBackbone=gpt-4o-mini, Prompt Setting=Baseline, Prompting Strategy=simple2025.10 | 69.6 | |
| Chain-of-ThoughtBackbone=Qwen2.5-14B-Instruct, Prompt Setting=Baseline, Prompting Strategy=cot2025.10 | 69.6 | |
| Chain-of-ThoughtBackbone=gpt-4o-mini, Prompt Setting=Baseline, Prompting Strategy=cot2025.10 | 69.4 | |
| Relevance Theory PromptingBackbone=Qwen2.5-7B-Instruct, Prompt Setting=Proposed, Prompting Strategy=relevance2025.10 | 68.3 | |
| Gricean PromptingBackbone=Qwen2.5-7B-Instruct, Prompt Setting=Proposed, Prompting Strategy=grice2025.10 | 66 | |
| Gricean PromptingBackbone=Llama-3.1-8B-Instruct, Prompt Setting=Proposed, Prompting Strategy=grice2025.10 | 65.6 | |
| Gricean PromptingBackbone=Llama-3.1-8B-Instruct, Prompt Setting=Short, Prompting Strategy=grice2025.10 | 63.5 | |
| Relevance Theory PromptingBackbone=Llama-3.1-8B-Instruct, Prompt Setting=Short, Prompting Strategy=relevance2025.10 | 62.3 | |
| Simple PromptingBackbone=Llama-3.1-8B-Instruct, Prompt Setting=Baseline, Prompting Strategy=simple2025.10 | 62.1 | |
| Chain-of-ThoughtBackbone=Qwen2.5-7B-Instruct, Prompt Setting=Baseline, Prompting Strategy=cot2025.10 | 61.2 | |
| Simple PromptingBackbone=Qwen2.5-7B-Instruct, Prompt Setting=Baseline, Prompting Strategy=simple2025.10 | 61 | |
| Relevance Theory PromptingBackbone=Llama-3.1-8B-Instruct, Prompt Setting=Proposed, Prompting Strategy=relevance2025.10 | 60.4 | |
| Relevance Theory PromptingBackbone=Qwen2.5-7B-Instruct, Prompt Setting=Short, Prompting Strategy=relevance2025.10 | 60.4 | |
| Chain-of-ThoughtBackbone=Llama-3.1-8B-Instruct, Prompt Setting=Baseline, Prompting Strategy=cot2025.10 | 59.2 | |
| Gricean PromptingBackbone=Qwen2.5-7B-Instruct, Prompt Setting=Short, Prompting Strategy=grice2025.10 | 59.2 |