Smart Help on AI2-THOR (Evaluation Environments)
59.3SRMCTSTG
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| MCTSTGnotes=Oracle/Upper Bound2024.04 | 59.3 | 80.4 | 61 | 67.8 | — | 16.271 | 54.1 | |
| BaseModel-PLλe=0.0, Learning=Progressive Learning2024.04 | 48.8 | 55.6 | 45.5 | 49.6 | -10.959 | 24.383 | 19.8 | |
| BaseModel-PL (Smart Helper)λe=1.0, Learning=Progressive Learning2024.04 | 48.3 | 54.8 | 49.8 | 50.6 | -10.625 | 24.65 | 20 | |
| BaseModel2024.04 | 41.9 | 51.5 | 40.8 | 41.2 | -15.179 | 24.945 | 17.6 | |
| BaseModel-w/o. CapabilityAblation=Without Capability Module2024.04 | 37.4 | 50.4 | 37.5 | 37.9 | -13.852 | 23.748 | 23.5 | |
| LLM2024.04 | 34.5 | 50.6 | 38.2 | 38.1 | — | 22.914 | 21 | |
| MCTS-heuristic2024.04 | 27.4 | 52.8 | 35.5 | 40 | — | 24.692 | 19.9 | |
| End2endλe=0.02024.04 | 26.7 | 44.1 | 14.9 | 31.5 | -23.218 | 24.343 | 21.2 | |
| BaseModelλe=1.02024.04 | 18.1 | 46.6 | 19.3 | 20 | -17.185 | 27.448 | 12.2 | |
| MCTS2024.04 | 17.8 | 48.2 | 26.6 | 32.8 | — | 26.743 | 12.6 | |
| Random2024.04 | 7.4 | 36.4 | 5.4 | 5.7 | -31.339 | 28.567 | 5.4 | |
| End2endλe=1.02024.04 | 6.7 | 37.2 | 5.7 | 5.7 | -26.183 | 28.726 | 4.9 | |
| MCTSRG2024.04 | 4.3 | 34.8 | 1.5 | 1.5 | — | 28.95 | 4.2 |