Pun Detection on MultiPun Homographic 1.0 (test)
100TPRQwen3-VL 30B-Instruct
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Qwen3-VL 30B-InstructType=Open-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 100 | -0.048 | 2.8 | 0.506 | 50.7 | 0.019 | |
| Claude Sonnet-4.5Type=Closed-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 99.2 | -0.012 | 10.2 | 0.11 | 52.4 | 0.065 | |
| Qwen3-VL 30B-A3B ThinkingType=Open-Source Reasoning-based VLMs, Prompt Strategy=biased-to-pun2026.04 | 99.2 | -0.112 | 7.8 | 0.39 | 51.7 | 0.036 | |
| Gemini-3-ProType=Closed-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 98.4 | -0.008 | 20.9 | 0.135 | 55.2 | 0.139 | |
| LLaVA-v1.6 Vicuna-13BType=Open-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 98 | -0.944 | 2.4 | 0.95 | 49.8 | 0.003 | |
| Qwen3-VL 8B-ThinkingType=Open-Source Reasoning-based VLMs, Prompt Strategy=biased-to-pun2026.04 | 98 | -0.215 | 4.8 | 0.554 | 50.5 | 0.016 | |
| Qwen3-VL 8B-InstructType=Open-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 96.8 | -0.263 | 14.7 | 0.351 | 52.7 | 0.082 | |
| GPT-5.1Type=Closed-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 95.6 | -0.036 | 24.3 | 0.201 | 55.1 | 0.146 | |
| GPT-4oType=Closed-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 95.6 | -0.004 | 21.1 | 0.122 | 54.1 | 0.121 | |
| GLM-4.1V 9B-ThinkingType=Open-Source Reasoning-based VLMs, Prompt Strategy=biased-to-pun2026.04 | 95.6 | -0.247 | 9.2 | 0.484 | 50.7 | 0.026 | |
| Llama-4 Scout-17BType=Open-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 91.2 | -0.275 | 34.1 | 0.408 | 56.5 | 0.193 |