High-level planning on VLABench vlm_evaluation 1.0 (test)
85.8AUCOracle
Evaluation Results
| Method | Links | |
|---|---|---|
| OraclePair=RynnBrain-Plan / GLM 4.1 T, Category=Heterogeneous Model Pairs, Sample Size=N=9002026.06 | 85.8 | |
| OraclePair=Qwen3-VL 8B I / T, Category=Homogeneous Open-Weight Pairs, Sample Size=20% subsample2026.06 | 81.92 | |
| OraclePair=GLM-4.6 I / T, Category=Homogeneous Open-Weight Pairs, Sample Size=N=9002026.06 | 81.77 | |
| OraclePair=Cosmos-R2 / Qwen3-VL 8B T, Category=Heterogeneous Model Pairs, Sample Size=20% subsample2026.06 | 79.97 | |
| OraclePair=GLM-4.6 I / 4.1 T, Category=Homogeneous Open-Weight Pairs, Sample Size=N=9002026.06 | 76.99 | |
| DIRECTPair=Qwen3-VL 8B I / T, Category=Homogeneous Open-Weight Pairs, Sample Size=20% subsample2026.06 | 59.57 | |
| DIRECTPair=Cosmos-R2 / Qwen3-VL 8B T, Category=Heterogeneous Model Pairs, Sample Size=20% subsample2026.06 | 57.32 | |
| DIRECTPair=GLM-4.6 I / 4.1 T, Category=Homogeneous Open-Weight Pairs, Sample Size=N=9002026.06 | 56.34 | |
| DIRECTPair=GLM-4.6 I / T, Category=Homogeneous Open-Weight Pairs, Sample Size=N=9002026.06 | 55.47 | |
| RandomPair=GLM-4.6 I / 4.1 T, Category=Homogeneous Open-Weight Pairs, Sample Size=N=9002026.06 | 48.22 | |
| Strongest-per-categoryPair=Qwen3-VL 8B I / T, Category=Homogeneous Open-Weight Pairs, Sample Size=20% subsample2026.06 | 48.22 | |
| Strongest-per-categoryPair=GLM-4.6 I / 4.1 T, Category=Homogeneous Open-Weight Pairs, Sample Size=N=9002026.06 | 47.95 | |
| Strongest-per-categoryPair=Cosmos-R2 / Qwen3-VL 8B T, Category=Heterogeneous Model Pairs, Sample Size=20% subsample2026.06 | 47.94 | |
| Strongest-per-categoryPair=GLM-4.6 I / T, Category=Homogeneous Open-Weight Pairs, Sample Size=N=9002026.06 | 46.6 | |
| DIRECTPair=RynnBrain-Plan / GLM 4.1 T, Category=Heterogeneous Model Pairs, Sample Size=N=9002026.06 | 46.32 | |
| OODPair=Cosmos-R2 / Qwen3-VL 8B T, Category=Heterogeneous Model Pairs, Sample Size=20% subsample2026.06 | 45.75 | |
| RandomPair=Qwen3-VL 8B I / T, Category=Homogeneous Open-Weight Pairs, Sample Size=20% subsample2026.06 | 45.64 | |
| RandomPair=GLM-4.6 I / T, Category=Homogeneous Open-Weight Pairs, Sample Size=N=9002026.06 | 44.57 | |
| OODPair=Qwen3-VL 8B I / T, Category=Homogeneous Open-Weight Pairs, Sample Size=20% subsample2026.06 | 44.27 | |
| RandomPair=Cosmos-R2 / Qwen3-VL 8B T, Category=Heterogeneous Model Pairs, Sample Size=20% subsample2026.06 | 43.62 | |
| OODPair=GLM-4.6 I / 4.1 T, Category=Homogeneous Open-Weight Pairs, Sample Size=N=9002026.06 | 42.07 | |
| OODPair=GLM-4.6 I / T, Category=Homogeneous Open-Weight Pairs, Sample Size=N=9002026.06 | 40.95 | |
| Strongest-per-categoryPair=RynnBrain-Plan / GLM 4.1 T, Category=Heterogeneous Model Pairs, Sample Size=N=9002026.06 | 36.05 | |
| RandomPair=RynnBrain-Plan / GLM 4.1 T, Category=Heterogeneous Model Pairs, Sample Size=N=9002026.06 | 32.95 | |
| OODPair=RynnBrain-Plan / GLM 4.1 T, Category=Heterogeneous Model Pairs, Sample Size=N=9002026.06 | 32.79 |