Video Question Answering on ETBench
43.6TVG F1Dispider
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| DispiderFrames=1 FPS, Inference Setting=Conventional video QA inference, Specialized Time Tokens=false2025.01 | 43.6 | 17.2 | 29.9 | 51.5 | 31.6 | 17.8 | 14.1 | 11.7 | |
| ETChatFrames=1 FPS, Inference Setting=Conventional video QA inference, Specialized Time Tokens=true2025.01 | 38.6 | 10.2 | 30.8 | 62.5 | 38.4 | 19.7 | 24.4 | 14.6 | |
| DispiderFrames=1 FPS, Inference Setting=Streaming video QA inference2025.01 | 36.1 | 15.5 | 27.3 | 54.2 | 33.8 | 18.9 | 18.8 | 12.4 | |
| TimeChatFrames=96, Inference Setting=Conventional video QA inference, Specialized Time Tokens=false2025.01 | 26.2 | 3.9 | 10.1 | 40.5 | 16.6 | 12.5 | 5.6 | 9.2 | |
| LITAFrames=100, Inference Setting=Conventional video QA inference, Specialized Time Tokens=true2025.01 | 22.2 | 4.6 | 18 | 23.9 | 39.7 | 17.2 | 21 | 12.2 | |
| VTG-LLMFrames=96, Inference Setting=Conventional video QA inference, Specialized Time Tokens=true2025.01 | 15.9 | 3.7 | 14.4 | 48.2 | 40.2 | 18.6 | 20.8 | 14.4 | |
| VideoLLM-OnlineFrames=2 FPS, Inference Setting=Streaming video QA inference2025.01 | 13.2 | 3.8 | 9.1 | 22.4 | 24 | 13.4 | 9.9 | 10.1 | |
| VTimeLLMFrames=100, Inference Setting=Conventional video QA inference, Specialized Time Tokens=false2025.01 | 7.6 | 1.9 | 18.2 | 28.9 | 12.4 | 13.1 | 8.7 | 6.4 | |
| VideoChatGPTFrames=100, Inference Setting=Conventional video QA inference, Specialized Time Tokens=false2025.01 | 7 | 1.3 | 15.1 | 28.8 | 8.8 | 11.3 | 5.7 | 10.2 | |
| Video-LLaVAFrames=8, Inference Setting=Conventional video QA inference, Specialized Time Tokens=false2025.01 | 7 | 1.9 | 15 | 28.9 | 28 | 15 | 0.9 | 8.3 | |
| PLLAVAFrames=16, Inference Setting=Conventional video QA inference, Specialized Time Tokens=false2025.01 | 6.9 | 1.1 | 5.7 | 28.9 | 13.3 | 10.6 | 9.7 | 11.8 | |
| LLaMA-VIDFrames=1 FPS, Inference Setting=Conventional video QA inference, Specialized Time Tokens=false2025.01 | 5.5 | 1.2 | 8 | 30 | 27.1 | 12.6 | 5.2 | 11.1 | |
| Video-LLaMA2Frames=8, Inference Setting=Conventional video QA inference, Specialized Time Tokens=false2025.01 | 0.1 | 0 | 0 | 1.5 | 0.6 | 14.5 | 0 | 15.2 |