World Modeling on Plancraft
98.4SmeltNeSyS
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| NeSySBackbone=Llama3.2-1B, Phase=Phase 2, Training Data=35%2026.02 | 98.4 | 95.1 | 93.7 | 67 | 87.7 | |
| GPT-5-miniBackbone=GPT-5-mini2026.02 | 93.8 | 91.5 | 60.3 | 47 | 73.8 | |
| NeSySBackbone=Qwen3-4B, Phase=Phase 12026.02 | 93.8 | 83 | 92.5 | 67.4 | 81.6 | |
| NeSySBackbone=Qwen3-4B, Phase=Phase 2, Training Data=35%2026.02 | 92.2 | 97.1 | 96.6 | 65.6 | 88.4 | |
| Llama3.1-8BBackbone=Llama3.1-8B2026.02 | 90.6 | 91.2 | 69.5 | 36.5 | 72.6 | |
| SFTBackbone=Qwen3-4B, Training Data=100%2026.02 | 87.5 | 99.8 | 94.8 | 70 | 90.1 | |
| Neural WMBackbone=Qwen3-4B, Phase=Phase 22026.02 | 87.5 | 100 | 93.1 | 51.7 | 85.1 | |
| Symbolic WMBackbone=Qwen3-4B, Phase=Phase 12026.02 | 85.9 | 79.3 | 82.2 | 66.1 | 76.9 | |
| Symbolic WMBackbone=Llama3.2-1B, Phase=Phase 22026.02 | 81.3 | 89.1 | 87.9 | 62.2 | 81.2 | |
| Symbolic WMBackbone=Qwen3-4B, Phase=Phase 22026.02 | 81.3 | 89.8 | 86.8 | 45.7 | 77 | |
| Phi-4-miniBackbone=Phi-4-mini2026.02 | 75 | 90.5 | 72.4 | 33 | 70.8 | |
| NeSySBackbone=Llama3.2-1B, Phase=Phase 12026.02 | 71.9 | 83.9 | 92.5 | 69.6 | 81 | |
| GPT-oss-20BBackbone=GPT-oss-20B2026.02 | 62.5 | 81.3 | 70.7 | 27.4 | 63.7 | |
| Neural WMBackbone=Qwen3-4B, Phase=Phase 12026.02 | 57.8 | 89.3 | 69.5 | 31.3 | 67.9 | |
| Qwen3-14BBackbone=Qwen3-14B2026.02 | 56.3 | 91.5 | 64.4 | 35.7 | 68.9 | |
| Neural WMBackbone=Llama3.2-1B, Phase=Phase 12026.02 | 54.7 | 88.1 | 69.5 | 28.7 | 66.4 | |
| Qwen3-8BBackbone=Qwen3-8B2026.02 | 48.4 | 90 | 68.4 | 33.9 | 68 | |
| Symbolic WMBackbone=Llama3.2-1B, Phase=Phase 12026.02 | 45.3 | 93.4 | 47.7 | 48.7 | 69.2 | |
| Neural WMBackbone=Llama3.2-1B, Phase=Phase 22026.02 | 34.4 | 97.8 | 75.3 | 47 | 75.4 | |
| SFTBackbone=Llama3.2-1B, Training Data=100%2026.02 | 31.3 | 98.8 | 73.6 | 67 | 80.5 |