Open-ended counting on TallyQA (test)
86.2Simple AccuracyPaLI-X-VPD
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| PaLI-X-VPDModel Scale=55B2023.12 | 86.2 | — | 76.6 | — | |
| PaLI-X-InstructModel Scale=55B2023.12 | 85.5 | — | 75.4 | — | |
| PaLI-3-VPDModel Scale=5B2023.12 | 83.1 | — | 70.9 | — | |
| Qwen-VLModel Scale=9.7B, Zero-shot=true2023.12 | 82.6 | — | 65.8 | — | |
| PaLI-3-InstructModel Scale=5B2023.12 | 81.9 | — | 70.4 | — | |
| Qwen-VL-ChatModel Scale=9.7B, Zero-shot=true2023.12 | 81.1 | — | 64 | — | |
| LLAVA-1.5Model Scale=Vicuna-13B, Zero-shot=true2023.12 | 76.9 | — | 65.4 | — | |
| InstructBLIPModel Scale=Vicuna-13B, Zero-shot=true2023.12 | 75.2 | — | 57.5 | — | |
| MoVieBackbone=X-101, #params (M)=88.8, FLOPs (G)=706.32020.04 | 74.9 | 1 | 56.8 | 1.43 | |
| TallyQABackbone=R-101+152, #params (M)=104.8, FLOPs (G)=1883.52020.04 | 71.8 | 1.13 | 56.2 | 1.43 | |
| MoVieBackbone=R-50, #params (M)=25.6, FLOPs (G)=176.12020.04 | 70.8 | 1.09 | 54.1 | 1.52 | |
| Count moduleBackbone=R-101, #params (M)=44.62020.04 | 70.5 | 1.15 | 50.9 | 1.58 | |
| TallyQA (FG-Only)Backbone=R-101, #params (M)=44.6, FLOPs (G)=1790.92020.04 | 69.4 | 1.18 | 51.8 | 1.5 | |
| MUTANBackbone=R-152, #params (M)=60.22020.04 | 56.5 | 1.51 | 49.1 | 1.59 |