Multi-choice Video Question Answering on MVBench
67.6Avg AccuracyTarsier-34B
Evaluation results
| Method | Links | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Tarsier-34BMode=Zero-shot2024.06 | 67.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoChat2-HD-F16ViEncoder=InternVideo2_S3-1B, LLM=Mistral-7B, Frames=162024.03 | 67.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoChat2-HDViEncoder=InternVideo2_S3-1B, LLM=Mistral-7B2024.03 | 65.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Tarsier-7BMode=Zero-shot2024.06 | 62.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoLLaMA2Model Scale=72B, #Frames=162024.06 | 62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoLLaMA2LLM size=72B, Video-Gen=false2024.12 | 62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVAction-7Bzero-shot=true2025.03 | 61.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| InternVideo2Mode=Zero-shot2024.06 | 60.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoChat2Mode=Zero-shot2024.06 | 60.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoChat2-7Bzero-shot=true2025.03 | 60.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoChat2ViEncoder=InternVideo2_S3-1B, LLM=Mistral-7B2024.03 | 60.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVA-OneVisionModel Scale=72B, #Frames=322024.06 | 59.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoGPT+2024.06 | 58.7 | 69 | 60 | 83 | 48.5 | 66.5 | 85.5 | 75.5 | 36 | 44 | 34 | 89.5 | 39.5 | 71 | 90.5 | 45 | 53 | 50 | 29.5 | 44 | 60 | |
| LLaVA-Video-7Bzero-shot=true2025.03 | 58.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PLLaVA 34BVision Encoder=ViT-L, LLM Size=34B2024.04 | 58.1 | 67.5 | 53 | 82 | 47 | 79 | 68.5 | 67.5 | 36.5 | 37.5 | 49.5 | 91 | 40.5 | 43 | 70 | 51.5 | 50 | 66.5 | 39.5 | 63.5 | 59 | |
| PLLAVAMode=Zero-shot2024.06 | 58.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PLLAVAModel Scale=34B, #Frames=162024.06 | 58.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoLLaMA2.1Model Scale=7B, #Frames=162024.06 | 57.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVA-OV-7Bzero-shot=true2025.03 | 56.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ST-LLMVision Encoder=BLIP2, LLM Size=7B2024.04 | 54.9 | 66 | 53.5 | 84 | 44 | 58.5 | 80.5 | 73.5 | 38.5 | 42.5 | 31 | 86.5 | 36.5 | 56.5 | 78.5 | 43 | 44.5 | 46.5 | 34.5 | 41.5 | 58.5 | |
| ST-LLMMode=Zero-shot2024.06 | 54.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoLLaMA2ViEncoder=CLIP-L-336, LLM=Mistral-7B2024.03 | 54.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoLLaMA2Model Scale=7B, #Frames=162024.06 | 54.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoLLaMA2LLM size=7B, Video-Gen=false2024.12 | 54.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoLLaMA2-7Bzero-shot=true2025.03 | 54.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| mPLUG-Owl3zero-shot=true2025.03 | 54.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoLLaMA2ViEncoder=CLIP-L-336, LLM=Mistral-8*7B2024.03 | 53.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoLLaMA2Model Scale=8x7B, #Frames=82024.06 | 53.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoLLaMA2LLM Size=46.7B, Vis Encoder=CLIP-L, Training Regime=Training-Based2026.02 | 53.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoLLaMA2Model Scale=7B, #Frames=82024.06 | 53.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Divot-LLMLLM size=7B, Video-Gen=true2024.12 | 52.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KTV-34B-normalLLM Size=34B, Vis Encoder=CLIP-L, Training Regime=Training-Free, Variant=normal2026.02 | 52.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KTV-34B-sparseLLM Size=34B, Vis Encoder=CLIP-L, Training Regime=Training-Free, Variant=sparse2026.02 | 51.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KTV-34B-denseLLM Size=34B, Vis Encoder=CLIP-L, Training Regime=Training-Free, Variant=dense2026.02 | 51.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoChat2Vision Encoder=UMT-L, LLM Size=7B2024.04 | 51.1 | 66 | 47.5 | 83.5 | 49.5 | 60 | 58 | 71.5 | 42.5 | 23 | 23 | 88.5 | 39 | 42 | 58.5 | 44 | 49 | 36.5 | 35 | 40.5 | 65.5 | |
| VideoChat22024.06 | 51.1 | 66 | 47.5 | 83.5 | 49.5 | 60 | 58 | 71.5 | 42.5 | 23 | 23 | 88.5 | 39 | 42 | 58.5 | 44 | 49 | 36.5 | 35 | 40.5 | 65.5 | |
| VideoChat2ViEncoder=UMT-L, LLM=Vicuna-7B2024.03 | 51.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoChat2Model Scale=7B, #Frames=162024.06 | 51.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoChat2LLM size=7B, Video-Gen=false2024.12 | 51.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PLLaVA 13BVision Encoder=ViT-L, LLM Size=13B2024.04 | 50.1 | 66 | 53 | 65.5 | 45 | 65 | 58 | 64.5 | 35.5 | 23.5 | 30 | 85 | 39.5 | 45.5 | 57 | 47.5 | 49.5 | 49 | 33 | 53 | 37 | |
| SF-LLaVA-7BLLM Size=34B, Vis Encoder=CLIP-L, Training Regime=Training-Free2026.02 | 49.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IG-VLMLLM Size=34B, Vis Encoder=CLIP-L, Training Regime=Training-Free2026.02 | 49 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PLLaVA 7BVision Encoder=ViT-L, LLM Size=7B2024.04 | 46.6 | 58 | 49 | 55.5 | 41 | 61 | 56 | 61 | 36 | 23.5 | 26 | 82 | 39.5 | 42 | 52 | 45 | 42 | 53.5 | 30.5 | 48 | 31 | |
| LLaVA-Next-VideoViEncoder=CLIP-L, LLM=Vicuna-7B2024.03 | 46.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVA-NeXT-VideoModel Scale=7B, #Frames=322024.06 | 46.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVA-NeXT-VideoLLM size=7B, Video-Gen=false2024.12 | 46.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KTV-7B-normalLLM Size=7B, Vis Encoder=CLIP-L, Training Regime=Training-Free, Variant=normal2026.02 | 46.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KTV-7B-sparseLLM Size=7B, Vis Encoder=CLIP-L, Training Regime=Training-Free, Variant=sparse2026.02 | 46.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KTV-7B-denseLLM Size=7B, Vis Encoder=CLIP-L, Training Regime=Training-Free, Variant=dense2026.02 | 46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT4-V2024.06 | 43.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT4-VVideo-Gen=false2024.12 | 43.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4VVision Encoder=N/A, LLM Size=GPT-42024.04 | 43.5 | 55.5 | 63.5 | 72 | 46.5 | 73.5 | 18.5 | 59 | 29.5 | 12 | 40.5 | 83.5 | 39 | 12 | 22.5 | 45 | 47.5 | 52 | 31 | 59 | 11 | |
| GPT-4V2024.06 | 43.5 | 55.5 | 63.5 | 72 | 46.5 | 73.5 | 18.5 | 59 | 29.5 | 12 | 40.5 | 83.5 | 39 | 12 | 22.5 | 45 | 47.5 | 52 | 31 | 59 | 11 | |
| GPT-4VLLM=GPT-42024.03 | 43.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4Vzero-shot=true2025.03 | 43.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SF-LLaVA-7BLLM Size=7B, Vis Encoder=CLIP-L, Training Regime=Training-Free2026.02 | 43.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IG-VLMLLM Size=7B, Vis Encoder=CLIP-L, Training Regime=Training-Free2026.02 | 42.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLAMA-VIDModel Scale=7B, #Frames=1 fps2024.06 | 41.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA-VIDLLM size=7B, Video-Gen=false2024.12 | 41.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Video-LLaVAModel Scale=7B, #Frames=82024.06 | 41 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Video-LLaVALLM size=7B, Video-Gen=false2024.12 | 41 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Video-LLaVALLM Size=7B, Vis Encoder=ViT-L, Training Regime=Training-Based2026.02 | 41 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Gemini 1.0 Pro2024.03 | 37.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VideoChatVision Encoder=CLIP-G, LLM Size=7B2024.04 | 35.5 | 33.5 | 26.5 | 56 | 33.5 | 40.5 | 53 | 40.5 | 30 | 25.5 | 27 | 48.5 | 35 | 20.5 | 42.5 | 46 | 26.5 | 41 | 23.5 | 23.5 | 36 | |
| VideoChat2024.06 | 35.5 | 33.5 | 26.5 | 56 | 33.5 | 40.5 | 53 | 40.5 | 30 | 25.5 | 27 | 48.5 | 35 | 20.5 | 42.5 | 46 | 26.5 | 41 | 23.5 | 23.5 | 36 | |
| Video-LLAMAVision Encoder=CLIP-G, LLM Size=7B2024.04 | 34.1 | 27.5 | 25.5 | 51 | 29 | 39 | 48 | 40.5 | 38 | 22.5 | 22.5 | 43 | 34 | 22.5 | 32.5 | 45.5 | 32.5 | 40 | 30 | 21 | 37 | |
| VideoLLaMA2024.06 | 34.1 | 27.5 | 25.5 | 51 | 29 | 39 | 48 | 40.5 | 38 | 22.5 | 22.5 | 43 | 34 | 22.5 | 32.5 | 45.5 | 32.5 | 40 | 30 | 21 | 37 | |
| Video-ChatGPTVision Encoder=ViT-L, LLM Size=7B2024.04 | 32.7 | 23.5 | 26 | 62 | 22.5 | 26.5 | 54 | 28 | 40 | 23 | 20 | 31 | 30.5 | 25.5 | 39.5 | 48.5 | 29 | 33 | 29.5 | 26 | 35.5 | |
| Video-ChatGPT2024.06 | 32.7 | 23.5 | 26 | 62 | 22.5 | 26.5 | 54 | 28 | 40 | 23 | 20 | 31 | 30.5 | 25.5 | 39.5 | 48.5 | 29 | 33 | 29.5 | 26 | 35.5 | |
| LLaMA-AdapterVision Encoder=ViT-B, LLM Size=7B2024.04 | 31.7 | 23 | 28 | 51 | 30 | 33 | 53.5 | 32.5 | 33.5 | 25.5 | 21.5 | 30.5 | 29 | 22.5 | 41.5 | 39.5 | 25 | 31.5 | 22.5 | 28 | 32 | |
| mPLUG-Owl-V2024.06 | 29.7 | 22 | 28 | 34 | 29 | 29 | 40.5 | 27 | 31.5 | 27 | 23 | 29 | 31.5 | 27 | 40 | 44 | 24 | 31 | 26 | 20.5 | 29.5 | |
| Random2024.06 | 27.3 | 25 | 25 | 33.3 | 25 | 25 | 33.3 | 25 | 33.3 | 25 | 25 | 25 | 33.3 | 25 | 33.3 | 33.3 | 25 | 33.3 | 25 | 20 | 30.9 | |
| Otter-V2024.06 | 26.8 | 23 | 23 | 27.5 | 27 | 29.5 | 53 | 28 | 33 | 24.5 | 23.5 | 27.5 | 26 | 28.5 | 18 | 38.5 | 22 | 22 | 23.5 | 19 | 19.5 |