State Tracking on Filesystem State Tracking Relative path-resolution
99Command-level AccuracyShellGames
Evaluation results
| Method | Links | |
|---|---|---|
| ShellGamesSequence Length=1002026.06 | 99 | |
| ShellGamesSequence Length=2002026.06 | 98.5 | |
| ShellGamesSequence Length=Mean2026.06 | 98.5 | |
| ShellGamesSequence Length=502026.06 | 98 | |
| GPT-5.4-miniSequence Length=502026.06 | 96 | |
| GPT-5.4-miniSequence Length=1002026.06 | 96 | |
| GPT-5.4-miniSequence Length=Mean2026.06 | 92 | |
| GPT-5.4-miniSequence Length=2002026.06 | 84 | |
| ShelLMSequence Length=502026.06 | 70 | |
| E-ShelLMSequence Length=502026.06 | 70 | |
| ShelLMSequence Length=1002026.06 | 60 | |
| E-ShelLMSequence Length=1002026.06 | 60 | |
| E-ShelLMSequence Length=Mean2026.06 | 53.3 | |
| ShelLMSequence Length=Mean2026.06 | 49 | |
| E-ShelLMSequence Length=2002026.06 | 30 | |
| ShelLMSequence Length=2002026.06 | 17 |