Command-to-action mapping on SCAN (length)
99.7AccuracyLeast-to-Most
Evaluation Results
| Method | Links | |
|---|---|---|
| Least-to-Mostmodel=code-davinci-0022022.05 | 99.7 | |
| vNQ¹constraint=hard constraint d(r) < inf2023.06 | 95.7 | |
| PModelconstraint=hard constraint d(r) < inf2023.06 | 91.72 | |
| Least-to-Mostmodel=text-davinci-002, sample_size=random subset of 100 commands2022.05 | 76 | |
| Least-to-Mostmodel=code-davinci-0012022.05 | 60.7 | |
| Standard promptingmodel=code-davinci-0022022.05 | 16.7 | |
| Chain-of-Thoughtmodel=code-davinci-0022022.05 | 16.2 | |
| Standard promptingmodel=text-davinci-002, sample_size=random subset of 100 commands2022.05 | 6 | |
| Standard promptingmodel=code-davinci-0012022.05 | 0.4 | |
| Chain-of-Thoughtmodel=text-davinci-002, sample_size=random subset of 100 commands2022.05 | 0 | |
| Chain-of-Thoughtmodel=code-davinci-0012022.05 | 0 |