Shell Command Correctness on ShellGames (test)
81.4Math AccuracyShellGames
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| ShellGamesinteraction_type=single-turn, strategy=Auto CoT2026.06 | 81.4 | 100 | 93.5 | 93.8 | 89.8 | |
| ShelLMinteraction_type=single-turn2026.06 | 78.6 | 93.8 | 91.1 | 62.5 | 84.8 | |
| E-ShelLMinteraction_type=single-turn2026.06 | 77.1 | 93.8 | 90.2 | 68.8 | 84.3 | |
| GPT-5.4-miniinteraction_type=single-turn2026.06 | 70 | 93.8 | 88.6 | 87.5 | 82.2 |