Science Simulation and Reasoning on ScienceWorld variations (test)
4Changes of State: Boiling SuccessSTARLING
Evaluation Results
| Method | Links | ||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| STARLINGModel Type=GRU, Param. Count=200K2024.06 | 4 | 5 | 10 | 0 | 19 | 11 | 10 | 13 | 11 | 9 | 14 | 25 | 94 | 25 | 25 | 12 | 9 | 22 | 30 | 14 | 48 | 35 | 31 | 21 | 18 | 12 | 9 | 27 | 18 | 16 | |
| DRRNModel Type=GRU, Param. Count=1.5M2024.06 | 3 | 4 | 1 | 3 | 10 | 8 | 6 | 13 | 10 | 7 | 20 | 26 | 56 | 19 | 19 | 9 | 16 | 20 | 29 | 11 | 48 | 47 | 31 | 9 | 10 | 13 | 13 | 13 | 19 | 17 | |
| Random-Valid2024.06 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 3 | 63 | 1 | 1 | 7 | 2 | 1 | 1 | 0 | 2 | 3 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 1 | |
| KG-A2CModel Type=GRU, Param. Count=5.5M2024.06 | 0 | 0 | 4 | 0 | 6 | 11 | 4 | 7 | 4 | 4 | 4 | 18 | 44 | 16 | 15 | 6 | 11 | 17 | 19 | 4 | 43 | 32 | 23 | 5 | 10 | 4 | 4 | 4 | 11 | 11 | |
| CALMModel Type=GPT2, Param. Count=131M2024.06 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 5 | 7 | 2 | 2 | 10 | 54 | 10 | 8 | 2 | 4 | 3 | 6 | 3 | 6 | 10 | 4 | 4 | 0 | 0 | 3 | 2 | 2 | 2 | |
| BCModel Type=T5 (base), Param. Count=11B, reliance on a valid action list during evaluation=true2024.06 | 0 | 0 | 1 | 0 | 4 | 1 | 1 | 3 | 2 | 5 | 4 | 29 | 19 | 17 | 21 | 8 | 3 | 6 | 16 | 5 | 26 | 14 | 2 | 4 | 2 | 5 | 5 | 4 | 6 | 13 | |
| TDTModel Type=T5 (base), Param. Count=11B2024.06 | 0 | 1 | 0 | 0 | 4 | 2 | 2 | 7 | 5 | 5 | 5 | 16 | 17 | 19 | 19 | 3 | 5 | 10 | 20 | 7 | 20 | 16 | 20 | 2 | 7 | 4 | 4 | 4 | 6 | 5 |