Environment Generation on LogicEnvEval
100Physics Pass Rate (Floor Plan)Holodeck
Evaluation Results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| HolodeckModel=DeepSeek-v32026.01 | 100 | 100 | 100 | 100 | 37.05 | 76 | 24 | 24 | 32 | 26.67 | 15.67 | |
| LogicEnvGenModel=DeepSeek-v32026.01 | 100 | 100 | 100 | 100 | 99.06 | 93.75 | 96 | 92 | 96 | 94.67 | 49.88 | |
| HolodeckModel=Gemini-2.5-Flash2026.01 | 100 | 100 | 100 | 100 | 37.05 | 68 | 32 | 24 | 20 | 25.33 | 17.88 | |
| LogicEnvGenModel=Gemini-2.5-Flash2026.01 | 100 | 100 | 100 | 100 | 96.47 | 92.78 | 96 | 92 | 92 | 93.33 | 50.96 | |
| HolodeckModel=Qwen2.5-72B2026.01 | 100 | 100 | 100 | 100 | 37.05 | 64 | 16 | 12 | 16 | 14.67 | 13.96 | |
| LogicEnvGenModel=Qwen2.5-72B2026.01 | 100 | 100 | 100 | 100 | 94.79 | 80.18 | 84 | 84 | 92 | 86.67 | 53.36 | |
| CoTModel=DeepSeek-v32026.01 | 96 | 40 | 20 | 52 | 63.38 | 85.71 | 64 | 56 | 56 | 58.67 | 29.16 | |
| IFGModel=DeepSeek-v32026.01 | 90.84 | 11 | 13.04 | 38.29 | 91.08 | 90.72 | 92 | 92 | 88 | 90.67 | 87.96 | |
| CoTModel=Gemini-2.5-Flash2026.01 | 86.46 | 0 | 4.17 | 30.21 | 86.1 | 79.78 | 84 | 88 | 68 | 80 | 55.2 | |
| IFGModel=Qwen2.5-72B2026.01 | 85.16 | 18.52 | 25.48 | 43.05 | 92.64 | 75.2 | 84 | 76 | 92 | 84 | 91.64 | |
| CoTModel=Qwen2.5-72B2026.01 | 84 | 16 | 26 | 42 | 65.47 | 73.77 | 60 | 56 | 52 | 56 | 37.6 | |
| IFGModel=Gemini-2.5-Flash2026.01 | 60.96 | 14.16 | 17.44 | 30.85 | 96.11 | 89.09 | 100 | 92 | 88 | 93.33 | 89.52 |