Pun Localization on MultiPun Homographic 1.0 (test)
100TPRQwen3-VL 30B-A3B Thinking
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Qwen3-VL 30B-A3B ThinkingType=Open-Source Reasoning-based VLMs, Prompt Strategy=biased-to-pun2026.04 | 100 | -0.008 | 15.1 | 0.319 | 54.1 | 0.135 | |
| Gemini-3-ProType=Closed-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 99.6 | -0.016 | 22.1 | 0.064 | 56.1 | 0.158 | |
| Claude Sonnet-4.5Type=Closed-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 99.6 | 0 | 4.6 | 0.052 | 51 | 0.028 | |
| Qwen3-VL 30B-InstructType=Open-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 99.6 | -0.155 | 5.2 | 0.275 | 51.2 | 0.033 | |
| Qwen3-VL 8B-ThinkingType=Open-Source Reasoning-based VLMs, Prompt Strategy=biased-to-pun2026.04 | 99.2 | -0.052 | 11.8 | 0.309 | 52.8 | 0.117 | |
| LLaVA-v1.6 Vicuna-13BType=Open-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 92.8 | -0.434 | 10.2 | 0.359 | 49.8 | 0.021 | |
| GPT-4oType=Closed-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 88.8 | -0.028 | 48 | 0.12 | 60.7 | 0.299 | |
| GPT-5.1Type=Closed-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 87.6 | -0.108 | 69.5 | 0.141 | 70.5 | 0.508 | |
| GLM-4.1V 9B-ThinkingType=Open-Source Reasoning-based VLMs, Prompt Strategy=biased-to-pun2026.04 | 84.1 | -0.175 | 55 | 0.052 | 61.3 | 0.411 | |
| Llama-4 Scout-17BType=Open-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 83.7 | 0.044 | 35.5 | -0.112 | 53.5 | 0.147 | |
| Qwen3-VL 8B-InstructType=Open-Source VLMs, Prompt Strategy=biased-to-pun2026.04 | 68.1 | -0.359 | 49 | 0.307 | 50.4 | 0.146 |