Scene Graph Generation on PSG v1.0 (test)
0Failure RateSceneGraphVLM
Evaluation results
| Method | Links | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SceneGraphVLMMode=SFT close2026.05 | 0 | 0.643 | 63.2 | 64.2 | 58.9 | 59.8 | 60.2 | 61.1 | 27.2 | 30.4 | 19.4 | 21.9 | 21.3 | 23.9 | 40.8 | 42.5 | |
| SceneGraphVLMMode=SFT + RL close2026.05 | 0 | 0.643 | 67.9 | 69 | 60.1 | 61.1 | 63 | 64 | 33.8 | 37.2 | 21.7 | 24.1 | 25.3 | 28 | 44.2 | 46 | |
| Gemini-3-flashMode=Base2026.05 | 0.09 | 4.671 | 31.2 | 49.7 | 30.3 | 50.9 | 29.2 | 47.7 | 11.6 | 15.6 | 12.8 | 17.4 | 11.7 | 15.8 | 20.4 | 31.8 | |
| R1-SGGMode=SFT+RL close2026.05 | 0.14 | 8.564 | 47.6 | 61 | 38 | 57.4 | 41 | 59.2 | 10.5 | 11.7 | 36.4 | 40.5 | 15.8 | 17.6 | 28.4 | 38.6 | |
| GPT-5-miniMode=Base2026.05 | 0.32 | 26.65 | 7.5 | 15.5 | 9.7 | 20.3 | 8 | 16.6 | 2 | 2.6 | 3.7 | 4.9 | 2.4 | 3.2 | 5.2 | 9.9 | |
| InternVL3.5-1B-Instruct (full finetune LLM head)Mode=SFT close2026.05 | 0.37 | 1.115 | 34.1 | 35 | 33.7 | 34.6 | 33.5 | 34.3 | 13.4 | 15.1 | 8.5 | 9.7 | 9.8 | 11.1 | 21.6 | 22.7 | |
| GPT-5.4-nanoMode=Base2026.05 | 0.64 | 2.456 | 7.5 | 14.8 | 5.9 | 12.4 | 6.2 | 12.6 | 1.3 | 1.7 | 0.7 | 0.9 | 0.8 | 1.1 | 3.5 | 6.9 | |
| DeepSeek-VL2-tiny-3BMode=Base2026.05 | 0.64 | 3.747 | 5.8 | 7.1 | 1.7 | 2.1 | 2.5 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 1.2 | 1.5 | |
| GPT-5.4-miniMode=Base2026.05 | 0.74 | 2.213 | 23.4 | 32.5 | 16.7 | 24.4 | 18.6 | 26.5 | 5.8 | 7.4 | 3.2 | 4.2 | 3.8 | 5 | 11.2 | 15.7 | |
| GPT-5-nanoMode=Base2026.05 | 0.78 | 32.639 | 7.8 | 13.8 | 6.3 | 11.6 | 6.5 | 11.7 | 1.1 | 1.5 | 0.7 | 0.9 | 0.8 | 1.1 | 3.6 | 6.4 | |
| InternVL3-1B (full finetune LLM head)Mode=SFT close2026.05 | 0.78 | 0.894 | 36 | 36.8 | 32.1 | 32.8 | 33.4 | 34.2 | 13.5 | 15 | 8 | 9 | 9.5 | 10.7 | 21.5 | 22.4 | |
| SceneGraphVLM (InternVL2.5-1B)Mode=SFT+RL close2026.05 | 1.15 | 0.704 | 61 | 62.4 | 44.6 | 45.6 | 50.6 | 51.7 | 22.2 | 27.5 | 11.4 | 14.5 | 14.3 | 18.2 | 32.5 | 34.9 | |
| InternVL3-1B (full finetune LLM head)Mode=SFT open2026.05 | 1.24 | 0.916 | 33.9 | 34.8 | 31.3 | 32.1 | 32.1 | 32.9 | 13.1 | 14.5 | 7.1 | 8 | 8.7 | 9.8 | 20.4 | 21.3 | |
| SceneGraphVLM (InternVL2.5-1B)Mode=SFT close2026.05 | 3.45 | 0.704 | 62.9 | 64.1 | 43.8 | 44.6 | 50.6 | 51.6 | 23 | 27.7 | 11.1 | 13.7 | 14.2 | 17.4 | 32.4 | 34.5 | |
| InternVL2.5-1BMode=Base2026.05 | 6.85 | 1.169 | 3.3 | 4.4 | 1 | 1.4 | 1.4 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0.7 | 1 | |
| Qwen3.5-4BMode=Base2026.05 | 7.58 | 3.042 | 39.9 | 47.4 | 21.7 | 27.1 | 26.5 | 32.3 | 8 | 10.3 | 3.7 | 4.7 | 4.5 | 5.8 | 15.5 | 19 | |
| InternVL3.5-1B-Instruct (LoRA r=16, α=32)Mode=SFT close2026.05 | 10.34 | 1.003 | 24.9 | 25.5 | 21.8 | 22.3 | 22.7 | 23.3 | 8 | 9.3 | 4 | 4.7 | 5 | 5.9 | 13.9 | 14.6 | |
| InternVL3-1BMode=Base2026.05 | 11.4 | 2.669 | 10.5 | 12.4 | 3.4 | 4.1 | 4.7 | 5.7 | 0.3 | 0.5 | 0.1 | 0.1 | 0.1 | 0.2 | 2.4 | 3 | |
| Qwen3.5-2BMode=Base2026.05 | 14.61 | 1.64 | 23 | 31.9 | 18.7 | 27.2 | 19.4 | 27.6 | 0.4 | 0.5 | 0.1 | 0.2 | 0.2 | 0.3 | 9.8 | 13.9 | |
| InternVL3.5-1BMode=Base2026.05 | 23.81 | 2.981 | 10.5 | 12.4 | 3.4 | 4.1 | 8.4 | 15.3 | 0.7 | 1.4 | 0.4 | 0.8 | 0.5 | 1 | 4.5 | 8.1 | |
| Qwen3-VL-4BMode=Base2026.05 | 45.37 | 7.778 | 36.2 | 37.9 | 7.8 | 8.6 | 12 | 12.9 | 0.7 | 0.9 | 0.2 | 0.3 | 0.3 | 0.4 | 6.1 | 6.7 | |
| Qwen3-VL-2BMode=Base2026.05 | 50.09 | 7.46 | 25.7 | 27.7 | 11 | 12.5 | 14.2 | 15.8 | 2 | 3.3 | 0.5 | 0.9 | 0.8 | 1.3 | 7.5 | 8.5 | |
| Qwen3.5-0.8BMode=Base2026.05 | 67.92 | 1.074 | 14 | 16.4 | 4.5 | 5.5 | 6.3 | 7.5 | 0 | 0.1 | 0 | 0 | 0 | 0 | 3.2 | 3.8 | |
| Ovis2.5-2B-InstructMode=Base2026.05 | 75.97 | 5.738 | 7 | 10.4 | 4.9 | 7.8 | 5.4 | 8.4 | 1.3 | 1.8 | 0.4 | 0.7 | 0.6 | 0.9 | 3 | 4.7 | |
| SmolVLM-500M-InstructMode=Base2026.05 | 86.08 | 2.871 | 0.3 | 0.3 | 0.1 | 0.1 | 0.1 | 0.1 | 0 | 0 | 0 | 0 | 0 | 0 | 0.1 | 0.1 |