Counting on TallyQA
82.2AccuracyGLM-4.6V
Evaluation results
| Method | Links | |
|---|---|---|
| GLM-4.6VModel Type=Open-Source & Baselines2026.05 | 82.2 | |
| MetaForgeTool Setting=w/ OOD Tools2026.06 | 81.17 | |
| Molmo2026.01 | 80.8 | |
| Gemini-2.5-FlashModel Type=Closed-Source2026.05 | 80.6 | |
| MetaForgeTool Setting=w/ IID Tools2026.06 | 80.52 | |
| Qwen2.5-VL2026.01 | 80 | |
| MAESTROModel Type=Ours2026.05 | 79.8 | |
| LLaVA-OneVision2026.01 | 79.7 | |
| PaliGemma2026.01 | 79.4 | |
| VTOOL-R1Model Type=Think with Images Methods2026.05 | 79.4 | |
| GPT-5Model Type=Closed-Source2026.05 | 79.2 | |
| Gemini-2.5-ProModel Type=Closed-Source2026.05 | 79 | |
| Qwen3-VL-32BModel Type=Open-Source & Baselines2026.05 | 78.6 | |
| Kimi-K2.5Model Type=Open-Source & Baselines2026.05 | 78.4 | |
| GPT-4oModel Type=Closed-Source2026.05 | 77.8 | |
| INTERNVL3-8B + PGTBackbone=INTERNVL3-8B, Data=PGT-augmented2026.05 | 77.4 | |
| INTERNVL3-8BBackbone=INTERNVL3-8B2026.05 | 77.2 | |
| Direct AnsweringModel Type=Open-Source & Baselines2026.05 | 74.8 | |
| Untrained ModelModel Type=Open-Source & Baselines2026.05 | 74.4 | |
| THINKLITE-VLMethod=THINKLITE-VL2026.05 | 73.9 | |
| QWEN2.5-VL-7B + PGTBackbone=QWEN2.5-VL-7B, Data=PGT-augmented2026.05 | 73.5 | |
| MathCoder-VLModel Type=Think with Images Methods2026.05 | 73.4 | |
| LLaVA-NeXT2026.01 | 73.3 | |
| ThymeModel Type=Think with Images Methods2026.05 | 73.2 | |
| DeepEyesModel Type=Think with Images Methods2026.05 | 73 | |
| GPT-5.4Tool Setting=w/ IID Tools2026.06 | 72.95 | |
| SmolVLM22026.01 | 72.9 | |
| VTS-VModel Type=Think with Images Methods2026.05 | 72.8 | |
| Chain-of-FocusModel Type=Think with Images Methods2026.05 | 72.6 | |
| QWEN2.5-VL-7BBackbone=QWEN2.5-VL-7B2026.05 | 72.4 | |
| QWEN2.5-VL-7B + SPECIALIZED MIXBackbone=QWEN2.5-VL-7B, Data=Specialized Mix2026.05 | 72.4 | |
| PixelReasonerModel Type=Think with Images Methods2026.05 | 71.8 | |
| LLaVA-AIMv2encoder=AIMv22026.01 | 71 | |
| Visual-ARFTModel Type=Think with Images Methods2026.05 | 70.8 | |
| LLaVA v1.52026.01 | 70.7 | |
| DeepEyes-v2Model Type=Think with Images Methods2026.05 | 70.6 | |
| IMAGE JIGSAWMethod=IMAGE JIGSAW2026.05 | 70.4 | |
| VisionReasonerModel Type=Think with Images Methods2026.05 | 70.2 | |
| LLaVA-AIMv2-2D-RoPEencoder=AIMv2, positional_embedding=2D-RoPE2026.01 | 69 | |
| LLaVA-SigLIP2encoder=SigLIP22026.01 | 68.4 | |
| QWEN2.5-VL-3B + PGTBackbone=QWEN2.5-VL-3B, Data=PGT-augmented2026.05 | 67.5 | |
| LLaVA-SigLIPencoder=SigLIP2026.01 | 67.2 | |
| QWEN2.5-VL-3BBackbone=QWEN2.5-VL-3B2026.05 | 66.4 | |
| LLaVA-2D-RoPEpositional_embedding=2D-RoPE2026.01 | 65.4 | |
| VIGORL-3BMethod=VIGORL-3B2026.05 | 64.8 | |
| LLaVA-SigLIP2-2D-RoPEencoder=SigLIP2, positional_embedding=2D-RoPE2026.01 | 64.6 | |
| LLAVA-NEXT-LLAMA3-8B + PGTBackbone=LLAVA-NEXT-LLAMA3-8B, Data=PGT-augmented2026.05 | 63.9 | |
| QWEN2.5-VL-3B + SPECIALIZED MIXBackbone=QWEN2.5-VL-3B, Data=Specialized Mix2026.05 | 63.4 | |
| LLaVA-SigLIP-2D-RoPEencoder=SigLIP, positional_embedding=2D-RoPE2026.01 | 61.6 | |
| LLAVA-NEXT-7B + PGTBackbone=LLAVA-NEXT-7B, Data=PGT-augmented2026.05 | 60.1 | |
| LLAVA-NEXT-LLAMA3-8BBackbone=LLAVA-NEXT-LLAMA3-8B2026.05 | 59.2 | |
| GPT-5-miniGrounded=true2025.12 | 55.4 | |
| GPT-5-miniGrounded=false2025.12 | 55.2 | |
| Gemma3-4b-it2026.01 | 52.5 | |
| VALOR2025.12 | 51 | |
| VALORModel Category=Open-Source, Tool Use=true2025.12 | 51 | |
| Claude-3.5-HaikuModel Category=Proprietary, Tool Use=true2025.12 | 50.1 | |
| Qwen3-8BModel Category=Open-Source, Tool Use=true2025.12 | 50.1 | |
| GPT-4oModel Category=Proprietary, Tool Use=true2025.12 | 49.9 | |
| VALOR-RLModel Category=Open-Source, Tool Use=true2025.12 | 49.5 | |
| Gemini-2.5-FlashModel Category=Proprietary, Tool Use=true2025.12 | 48.9 | |
| Gemma-3-12BModel Category=Open-Source, Tool Use=true2025.12 | 48.9 | |
| o4-miniModel Category=Proprietary, Tool Use=true2025.12 | 47.9 | |
| Llama-3.2-11BModel Category=Open-Source, Tool Use=true2025.12 | 47.7 | |
| LLAVA-NEXT-7BBackbone=LLAVA-NEXT-7B2026.05 | 45.2 | |
| Gemini-2.0-FlashModel Category=Proprietary, Tool Use=true2025.12 | 43.6 | |
| SPATIAL-LADDER-3BMethod=SPATIAL-LADDER-3B2026.05 | 43.1 |