World Modeling on Webshop (test)
100SearchSymbolic WM
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Symbolic WMBackbone=Llama3.2-1B, Phase=12026.02 | 100 | 23.5 | 81.6 | 83.8 | 75 | 96.5 | 83.4 | |
| Symbolic WMBackbone=Llama3.2-1B, Phase=22026.02 | 100 | 88.2 | 98 | 83.8 | 100 | 96.5 | 91.5 | |
| NeSySBackbone=Llama3.2-1B, Phase=2, Training Data Scale=60%2026.02 | 100 | 91.2 | 95.9 | 83.8 | 100 | 100 | 92.2 | |
| Symbolic WMBackbone=Qwen3-4B, Phase=12026.02 | 100 | 23.5 | 73.5 | 66.6 | 33 | 97.2 | 68.9 | |
| Symbolic WMBackbone=Qwen3-4B, Phase=22026.02 | 100 | 100 | 100 | 83.8 | 95.5 | 92.9 | 90.8 | |
| NeSySBackbone=Qwen3-4B, Phase=2, Training Data Scale=60%2026.02 | 100 | 100 | 95.9 | 83.8 | 100 | 100 | 92.6 | |
| NeSySBackbone=Llama3.2-1B, Phase=12026.02 | 98.5 | 2.9 | 73.5 | 83.8 | 71.4 | 95 | 80.9 | |
| NeSySBackbone=Qwen3-4B, Phase=12026.02 | 97 | 2.9 | 65.3 | 83.8 | 29.5 | 99.3 | 74.4 | |
| GPT-5-miniType=Baseline Model2026.02 | 89.4 | 94.1 | 81.6 | 66.9 | 94.6 | 96.5 | 81.4 | |
| Llama3.1-8BType=Baseline Model2026.02 | 0 | 2.9 | 10.2 | 50 | 2.7 | 29.1 | 28.9 | |
| Qwen3-8BType=Baseline Model2026.02 | 0 | 0 | 14.3 | 39.5 | 2.7 | 32.6 | 25.1 | |
| Qwen3-14BType=Baseline Model2026.02 | 0 | 2.9 | 12.2 | 46.2 | 2.7 | 31.9 | 27.9 | |
| Phi-4-miniType=Baseline Model2026.02 | 0 | 0 | 4.1 | 50 | 2.7 | 11.3 | 24.9 | |
| GPT-oss-20BType=Baseline Model2026.02 | 0 | 0 | 16.3 | 50 | 5.4 | 0.7 | 24 | |
| SFTBackbone=Llama3.2-1B, Training Data Scale=100%2026.02 | 0 | 58.8 | 20.4 | 50 | 10.7 | 100 | 47.5 | |
| Neural WMBackbone=Llama3.2-1B, Phase=12026.02 | 0 | 0 | 0 | 50 | 0.9 | 0.7 | 22.2 | |
| Neural WMBackbone=Llama3.2-1B, Phase=22026.02 | 0 | 52.9 | 19.2 | 50 | 12.4 | 100 | 45.9 | |
| SFTBackbone=Qwen3-4B, Training Data Scale=100%2026.02 | 0 | 64.7 | 18.4 | 52.5 | 11.6 | 92.2 | 47.3 | |
| Neural WMBackbone=Qwen3-4B, Phase=12026.02 | 0 | 0 | 16.3 | 50 | 3.6 | 27 | 28.9 | |
| Neural WMBackbone=Qwen3-4B, Phase=22026.02 | 0 | 64.7 | 19.2 | 51 | 13.5 | 93.1 | 46.1 |