Video Captioning on E.T. Bench-Captioning (test)
46.9DVC F1GPT-4o
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| GPT-4oModel category=Proprietary, Tested on subset=false2025.01 | 46.9 | 22.3 | 23.1 | 14.9 | 35 | 18.6 | |
| Tarsier2-7BModel category=Open-source (<=10B), Fine-tuned on E.T. Instruct 164K=true, Tested on subset=false2025.01 | 46.5 | 28.8 | 24.6 | 16.4 | 35.5 | 22.6 | |
| Qwen2-VL-7BModel category=Open-source (<=10B), Fine-tuned on E.T. Instruct 164K=true, Tested on subset=true, Evaluation protocol=Official checkpoint and inference code2025.01 | 44.3 | 25.3 | 25.7 | 15.6 | 35 | 20.4 | |
| Tarsier-7BModel category=Open-source (<=10B), Fine-tuned on E.T. Instruct 164K=true, Tested on subset=true2025.01 | 42.8 | 19.1 | 23.7 | 15.2 | 33.2 | 17.1 | |
| LLAVA-OV-72BModel category=Open-source (>10B), Tested on subset=false, Evaluation protocol=Official checkpoint and inference code2025.01 | 41.9 | 16.3 | 25.6 | 13.9 | 33.8 | 15.1 | |
| E.T. ChatModel category=Open-source (<=10B), Fine-tuned on E.T. Instruct 164K=true, Tested on subset=false2025.01 | 38.4 | 19.7 | 24.4 | 14.6 | 31.4 | 17.1 | |
| LLaVA-Video-72BModel category=Open-source (>10B), Tested on subset=true, Evaluation protocol=Official checkpoint and inference code2025.01 | 37 | 15.7 | 20.4 | 13.5 | 28.7 | 14.6 | |
| Gemini-1.5-FlashModel category=Proprietary, Tested on subset=true2025.01 | 31.6 | 14.9 | 16.5 | 13.3 | 24.1 | 14.1 | |
| Video-LLaVA-7BModel category=Open-source (<=10B), Tested on subset=true2025.01 | 28 | 15 | 0.9 | 8.3 | 14.4 | 11.7 | |
| Gemini-1.5-ProModel category=Proprietary, Tested on subset=true2025.01 | 24 | 17.5 | 5.8 | 9.8 | 14.9 | 13.7 | |
| LLaVA-OV-7BModel category=Open-source (<=10B), Tested on subset=false, Evaluation protocol=Official checkpoint and inference code2025.01 | 22 | 15.1 | 9.5 | 10.6 | 15.8 | 12.8 | |
| LLaVA-Video-7BModel category=Open-source (<=10B), Tested on subset=false, Evaluation protocol=Official checkpoint and inference code2025.01 | 20.6 | 14.7 | 6.5 | 13.1 | 13.6 | 14.1 | |
| GPT-4VModel category=Proprietary, Tested on subset=true2025.01 | 16.1 | 19.4 | 21.9 | 13.5 | 19 | 16.4 | |
| Qwen2-VL-72BModel category=Open-source (>10B), Tested on subset=true, Evaluation protocol=Official checkpoint and inference code2025.01 | 15.3 | 13.9 | 11 | 12.8 | 13.2 | 13.4 | |
| PLLAVA-34BModel category=Open-source (>10B), Tested on subset=true2025.01 | 13.3 | 10.6 | 9.7 | 11.8 | 11.5 | 11.2 | |
| VideoLLaMA2-7BModel category=Open-source (<=10B), Tested on subset=true2025.01 | 0.6 | 14.5 | 0 | 15.2 | 0.3 | 14.8 |