Long Video Understanding on Video-MME long 1.0
67.4Accuracy (No Subs)Gemini 1.5 Pro
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Gemini 1.5 ProFrames=0.5 FPS2024.06 | 67.4 | 77.4 | |
| Gemini 1.5 Pro2024.12 | 67.4 | 77.4 | |
| Gemini-1.5-Pro2024.09 | 67.4 | 77.4 | |
| GPT-4oFrames=3842024.06 | 65.3 | 72.1 | |
| GPT4-O2024.12 | 65.3 | 72.1 | |
| GPT-4o2024.09 | 65.3 | 72.1 | |
| LinVT-Qwen2-VLSize=7B2024.12 | 63.1 | 63.3 | |
| Qwen2-VL-72BFrames=7682024.06 | 62.2 | 74.3 | |
| Gemini 1.5 FlashFrames=0.5 FPS2024.06 | 61.1 | 68.8 | |
| LinVT-InternVL2Size=8B2024.12 | 60.2 | 63.5 | |
| LLaVA-OneVision-72BFrames=322024.06 | 60 | 62.4 | |
| LLaVA-OneVisionSize=72B2024.09 | 60 | 62.4 | |
| Oryx-1.5Size=32B2024.09 | 59.1 | 69.7 | |
| GPT-4o miniFrames=2502024.06 | 58.6 | 63.4 | |
| LinVT-BLIP-3Size=4B2024.12 | 58.3 | 62.4 | |
| LinVT-MolmoSize=7B2024.12 | 57.6 | 61.2 | |
| OryxSize=34B2024.09 | 53.9 | 58 | |
| VILA-1.5Size=40B2024.09 | 53.8 | 55.7 | |
| GPT-4VFrames=102024.06 | 53.5 | 56.9 | |
| GPT4-V2024.12 | 53.5 | 56.9 | |
| GPT-4V2024.09 | 53.5 | 56.9 | |
| LinVT-AquilaSize=2B2024.12 | 53.4 | 55.3 | |
| DrVideoFrames=0.2 FPS2024.06 | 51.7 | 71.7 | |
| Claude 3.5 SonnetFrames=202024.06 | 51.2 | 54.7 | |
| Oryx-1.5Size=7B2024.09 | 51.2 | 58.3 | |
| OryxSize=7B2024.12 | 50.3 | 55.8 | |
| OryxSize=7B2024.09 | 50.3 | 55.8 | |
| VITA-8x7BFrames=322024.06 | 48.6 | 50.9 | |
| VideoCCAMSize=14B2024.12 | 46.7 | 49.9 | |
| VideoCCAMSize=14B2024.09 | 46.7 | 49.9 | |
| KangarooSize=8B2024.12 | 46.6 | 49.3 | |
| KangarooSize=8B2024.09 | 46.6 | 49.3 | |
| LongVAFrames=1282024.06 | 46.2 | 47.6 | |
| LongVASize=7B2024.12 | 46.2 | 47.6 | |
| LongVASize=7B2024.09 | 46.2 | 47.6 | |
| LLoViFrames=0.2 FPS, re-implemented=true2024.06 | 45.4 | 67.7 | |
| LinVT-MiphaSize=1.6B2024.12 | 44.5 | 46.1 | |
| VideoLLaMA2Size=7B2024.12 | 42.1 | 43.8 | |
| VideoLLaMA2Size=7B2024.09 | 42.1 | 43.8 | |
| VideoAgentFrames=7.6, re-implemented=true2024.06 | 40.2 | 44.4 | |
| Video-LLaVASize=7B2024.12 | 39.9 | 41.6 | |
| LongVILASize=8B2024.12 | 39.7 | — | |
| LongVILASize=8B2024.09 | 39.7 | — | |
| Qwen-VL-ChatSize=7B2024.12 | 37.8 | 37.9 | |
| ShareGPT4VideoSize=8B2024.12 | 35 | 37.9 | |
| DrVideo (only subs)Frames=0.2 FPS, subtitles_only=true2024.06 | — | 68.5 |