Grounded Multi-Video Question Answering on MAGMaR (test)
82.2Reference Precision (Ref-P)CRAFT
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| CRAFTBase VLM=Qwen3.5-9B, Frame Sampling=DKS, Number of Frames=642026.05 | 82.2 | 74.3 | 77.2 | 92.7 | 45.3 | 57.4 | 71.5 | |
| CRAFTBase VLM=Qwen3.5-9B, Atomic Claims=true, Number of Frames=1282026.05 | 80.8 | 76.2 | 76.4 | 94.4 | 33.6 | 42.6 | 67.3 | |
| CRAFTBase VLM=Qwen3.5-9B, Frame Sampling=uniform, Number of Frames=642026.05 | 77.5 | 77.5 | 76.9 | 90.2 | 50.3 | 61.6 | 72.3 | |
| InternVL-3.5-30B-A3BNumber of Frames=128, ASR access=true2026.05 | 76.1 | 72.2 | 74.1 | 65.9 | 55.1 | 60 | 67.2 | |
| CRAFTBase VLM=Qwen3.5-9B, ASR access=true, Number of Frames=1282026.05 | 76 | 81 | 78.3 | 93.5 | 51.2 | 63.5 | 73.9 | |
| InternVL-3.5-30B-A3BNumber of Frames=128, ASR access=false2026.05 | 74.9 | 68.8 | 71.7 | 64.5 | 52.1 | 57.6 | 64.9 | |
| Gemma-4-31BNumber of Frames=128, ASR access=true2026.05 | 71.2 | 70.1 | 70.6 | 60.1 | 56.1 | 58 | 64.4 | |
| Gemma-4-31BNumber of Frames=128, ASR access=false2026.05 | 70.1 | 65.8 | 67.9 | 58.9 | 53.2 | 55.9 | 62 | |
| Molmo2-8BNumber of Frames=1282026.05 | 62.3 | 54.1 | 57.9 | 49.8 | 42.1 | 45.7 | 51.8 | |
| CRAFTBase VLM=Qwen3.5-9B, Critic Loop=true, Number of Frames=1282026.05 | 49.1 | 76.6 | 48 | 85.4 | 25.9 | 36 | 53.5 | |
| CRAFTBase VLM=Qwen3.5-9B, Configuration=Baseline, Number of Frames=1282026.05 | 43.7 | 75.6 | 43 | 87.5 | 25.1 | 35.9 | 51.8 |