Language-conditioned Manipulation Planning on VLABench
64.83Success RateDIRECT
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| DIRECTModel Pair=Gemini 3 Flash Minimal / Thinking, Category=Homogeneous Closed-Weight Pairs2026.06 | 64.83 | 8.47 | 57.99 | |
| Gemini 3 Flash ThinkingModel Pair=Gemini 3 Flash Minimal / Thinking, Role=Expensive Model, Category=Homogeneous Closed-Weight Pairs2026.06 | 64.22 | 15.86 | — | |
| DIRECTModel Pair=Gemini-ER 1.6 Minimal / Thinking, Category=Homogeneous Closed-Weight Pairs2026.06 | 63.92 | 4.72 | 37.09 | |
| OOD Detection RoutingModel Pair=Gemini-ER 1.6 Minimal / Thinking, Category=Homogeneous Closed-Weight Pairs2026.06 | 63.19 | 3.13 | 19.68 | |
| Gemini-ER 1.6 MinimalModel Pair=Gemini-ER 1.6 Minimal / Thinking, Role=Cheap Model, Category=Homogeneous Closed-Weight Pairs2026.06 | 62.88 | 2.31 | — | |
| Gemini-ER 1.6 ThinkingModel Pair=Gemini-ER 1.6 Minimal / Thinking, Role=Expensive Model, Category=Homogeneous Closed-Weight Pairs2026.06 | 62.62 | 7.02 | — | |
| Random RoutingModel Pair=Gemini-ER 1.6 Minimal / Thinking, Category=Homogeneous Closed-Weight Pairs2026.06 | 62.46 | 4.67 | 0 | |
| Random RoutingModel Pair=Gemini 3 Flash Minimal / Thinking, Category=Homogeneous Closed-Weight Pairs2026.06 | 62.11 | 8.82 | 15.53 | |
| OOD Detection RoutingModel Pair=Gemini 3 Flash Minimal / Thinking, Category=Homogeneous Closed-Weight Pairs2026.06 | 61.83 | 6.7 | 10.81 | |
| Gemini 3 Flash MinimalModel Pair=Gemini 3 Flash Minimal / Thinking, Role=Cheap Model, Category=Homogeneous Closed-Weight Pairs2026.06 | 61.22 | 1.88 | — | |
| DIRECTModel Pair=RynnBrain-Plan / GLM 4.1 Thinking, Category=Heterogeneous Model Pairs2026.06 | 54.33 | 27.5 | 63.94 | |
| GLM 4.1 ThinkingModel Pair=GLM-4.6 Instruct / 4.1 Thinking, Role=Expensive Model, Category=Homogeneous Open-Weight Pairs2026.06 | 54.02 | 45.1 | — | |
| GLM 4.1 ThinkingModel Pair=RynnBrain-Plan / GLM 4.1 Thinking, Role=Expensive Model, Category=Heterogeneous Model Pairs2026.06 | 54.02 | 45.1 | — | |
| OOD Detection RoutingModel Pair=RynnBrain-Plan / GLM 4.1 Thinking, Category=Heterogeneous Model Pairs2026.06 | 52.56 | 32.72 | 50.42 | |
| DIRECTModel Pair=GLM-4.6 Instruct / 4.1 Thinking, Category=Homogeneous Open-Weight Pairs2026.06 | 51.79 | 31.82 | 75.26 | |
| OOD Detection RoutingModel Pair=GLM-4.6 Instruct / 4.1 Thinking, Category=Homogeneous Open-Weight Pairs2026.06 | 49.83 | 39.6 | 54.52 | |
| Random RoutingModel Pair=RynnBrain-Plan / GLM 4.1 Thinking, Category=Heterogeneous Model Pairs2026.06 | 49.44 | 22.17 | 32.51 | |
| Qwen3-VL 8B ThinkingModel Pair=Qwen3-VL 8B Instruct / Thinking, Role=Expensive Model, Category=Homogeneous Open-Weight Pairs, Subsampled=20%2026.06 | 48.28 | 120.76 | — | |
| Qwen3-VL 8B ThinkingModel Pair=Cosmos-R2 / Qwen3-VL 8B Thinking, Role=Expensive Model, Category=Heterogeneous Model Pairs, Subsampled=20%2026.06 | 48.28 | 120.76 | — | |
| DIRECTModel Pair=Qwen3-VL 8B Instruct / Thinking, Category=Homogeneous Open-Weight Pairs, Subsampled=20%2026.06 | 47.48 | 81.72 | 74.78 | |
| DIRECTModel Pair=Cosmos-R2 / Qwen3-VL 8B Thinking, Category=Heterogeneous Model Pairs, Subsampled=20%2026.06 | 46.98 | 86.36 | 71.16 | |
| OOD Detection RoutingModel Pair=Qwen3-VL 8B Instruct / Thinking, Category=Homogeneous Open-Weight Pairs, Subsampled=20%2026.06 | 46.89 | 106.22 | 54.2 | |
| GLM-4.6 ThinkingModel Pair=GLM-4.6 Instruct / Thinking, Role=Expensive Model, Category=Homogeneous Open-Weight Pairs2026.06 | 46.09 | 24.08 | — | |
| RynnBrain-PlanModel Pair=RynnBrain-Plan / GLM 4.1 Thinking, Role=Cheap Model, Category=Heterogeneous Model Pairs2026.06 | 45.25 | 1.67 | — | |
| DIRECTModel Pair=GLM-4.6 Instruct / Thinking, Category=Homogeneous Open-Weight Pairs2026.06 | 44.69 | 16.97 | 74.92 | |
| OOD Detection RoutingModel Pair=GLM-4.6 Instruct / Thinking, Category=Homogeneous Open-Weight Pairs2026.06 | 42.73 | 20.85 | 55.34 | |
| Random RoutingModel Pair=Qwen3-VL 8B Instruct / Thinking, Category=Homogeneous Open-Weight Pairs, Subsampled=20%2026.06 | 40.62 | 64.4 | 48.48 | |
| Random RoutingModel Pair=GLM-4.6 Instruct / 4.1 Thinking, Category=Homogeneous Open-Weight Pairs2026.06 | 40.02 | 22.39 | 46.33 | |
| Random RoutingModel Pair=Cosmos-R2 / Qwen3-VL 8B Thinking, Category=Heterogeneous Model Pairs, Subsampled=20%2026.06 | 38.93 | 64.4 | 45.03 | |
| OOD Detection RoutingModel Pair=Cosmos-R2 / Qwen3-VL 8B Thinking, Category=Heterogeneous Model Pairs, Subsampled=20%2026.06 | 37.32 | 49.37 | 38.47 | |
| Random RoutingModel Pair=GLM-4.6 Instruct / Thinking, Category=Homogeneous Open-Weight Pairs2026.06 | 35.97 | 13.24 | 43.68 | |
| Qwen3-VL 8B InstructModel Pair=Qwen3-VL 8B Instruct / Thinking, Role=Cheap Model, Category=Homogeneous Open-Weight Pairs, Subsampled=20%2026.06 | 31.64 | 1.85 | — | |
| Cosmos-R2Model Pair=Cosmos-R2 / Qwen3-VL 8B Thinking, Role=Cheap Model, Category=Heterogeneous Model Pairs, Subsampled=20%2026.06 | 29.56 | 2.02 | — | |
| GLM-4.6 InstructModel Pair=GLM-4.6 Instruct / 4.1 Thinking, Role=Cheap Model, Category=Homogeneous Open-Weight Pairs2026.06 | 27.24 | 2.11 | — | |
| GLM-4.6 InstructModel Pair=GLM-4.6 Instruct / Thinking, Role=Cheap Model, Category=Homogeneous Open-Weight Pairs2026.06 | 27.24 | 2.11 | — |