Pairwise evaluator agreement with human judgment on Procthor-10K Complex 1.0 (test)
79.5Method A SceneCritic ScoreSceneCritic
Evaluation results
| Method | Links | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SceneCriticMethod A=holodeck, Method B=layoutgpt, Difficulty=Complex2026.04 | 79.5 | 95 | 68.3 | 100 | 100 | 0 | 87.5 | 12.5 | 93.8 | 6.2 | 100 | 87.5 | 93.8 | 1 | 0.53 | 0.77 | — | — | — | — | — | — | |
| SceneCriticMethod A=holodeck, Method B=layoutvlm, Difficulty=Complex2026.04 | 79.5 | 95 | 94 | 95 | 6.2 | 93.8 | 6.2 | 93.8 | 6.2 | 93.8 | 93.8 | 93.8 | 93.8 | 0.787 | 0.787 | 0.77 | — | — | — | — | — | — | |
| SceneCriticMethod A=heuristic, Method B=holodeck, Difficulty=Complex2026.04 | 79.39 | 75 | 79.5 | 95 | 81.2 | 18.8 | 81.2 | 18.8 | 87.5 | 12.5 | 81.2 | 81.2 | 87.5 | 0.426 | 0.383 | 0.563 | — | — | — | — | — | — | |
| SceneCriticMethod A=heuristic, Method B=layoutgpt, Difficulty=Complex2026.04 | 79.39 | 75 | 68.3 | 100 | 87.5 | 12.5 | 87.5 | 12.5 | 100 | 0 | 87.5 | 87.5 | 100 | 0.562 | 0.53 | 1 | — | — | — | — | — | — | |
| SceneCriticMethod A=heuristic, Method B=layoutvlm, Difficulty=Complex2026.04 | 79.39 | 75 | 94 | 95 | 18.8 | 81.2 | 6.2 | 93.8 | 12.5 | 87.5 | 81.2 | 93.8 | 87.5 | 0.426 | 0.787 | 0.563 | — | — | — | — | — | — | |
| SceneCriticMethod A=layoutgpt, Method B=layoutvlm, Difficulty=Complex2026.04 | 68.3 | 100 | 94 | 95 | 0 | 100 | 6.2 | 93.8 | 0 | 100 | 100 | 93.8 | 100 | 1 | 0.787 | 1 | — | — | — | — | — | — |