Scientific Reasoning in Text-based Environments on ScienceWorld (test)
44.8Task 1-1 ScoreExpert
Evaluation Results
| Method | Links | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ExpertBackbone=flan-t5-large, Evaluation Protocol=fine-tuned2024.02 | 44.8 | 30.7 | 8.7 | 5.8 | 73.8 | 72 | 100 | 96.7 | 28.5 | 17 | 22.9 | 13.7 | 85 | 69.9 | 8 | 36.6 | 41.5 | 66.5 | 16.9 | 17 | — | |
| SayCanpBase Agent Strategy=SayCan, PANDA Enhancement=true2024.02 | 1 | 1.8 | 27.2 | 26.3 | 25 | 46.7 | 14.3 | 80.7 | 8.4 | 50.1 | 18.1 | 8.6 | 54 | 48.8 | 12.8 | 9.2 | 24.1 | 18 | 33.5 | 32.4 | 10 | |
| SayCanBase Agent Strategy=SayCan2024.02 | 0.7 | 2 | 27.9 | 29.5 | 32 | 40.7 | 10.7 | 79.8 | 8.3 | 42.8 | 17.4 | 7.8 | 62 | 50.6 | 12.6 | 9.1 | 21.6 | 13.8 | 29.9 | 25.3 | — | |
| ReActpBase Agent Strategy=ReAct, PANDA Enhancement=true2024.02 | 0.2 | 0 | 9.6 | 37.7 | 22.8 | 76.3 | 26.6 | 72.5 | 5.5 | 23.7 | 26.4 | 10.4 | 95.5 | 81 | 7 | 2.3 | 21.7 | 12.8 | 38.9 | 2.1 | 16 | |
| ReflexionpBase Agent Strategy=Reflexion, PANDA Enhancement=true2024.02 | 0.1 | 0 | 9.2 | 77.6 | 19.6 | 75.5 | 28.4 | 85 | 6.3 | 49.4 | 27.3 | 11.5 | 72 | 75.2 | 6.4 | 6.2 | 38.5 | 18.9 | 39.4 | 10.2 | 9 | |
| ReActBase Agent Strategy=ReAct2024.02 | 0 | 0.1 | 11.3 | 33.3 | 15.4 | 64.8 | 15.1 | 64.1 | 3.7 | 36.2 | 18 | 7.2 | 50 | 46.4 | 4 | 0 | 16.5 | 10 | 21.7 | 0.2 | — | |
| ReflexionBase Agent Strategy=Reflexion2024.02 | 0 | 0 | 22.2 | 73.8 | 27.4 | 85.1 | 16.4 | 81.4 | 6 | 52.5 | 29.3 | 10.4 | 82.5 | 61.4 | 6.2 | 2.4 | 31.7 | 14.4 | 44.6 | 4.3 | — |