State Tracking on Filesystem State Tracking Absolute path-resolution
100Command-level AccuracyGPT-5.4-mini
Evaluation results
| Method | Links | |
|---|---|---|
| GPT-5.4-miniSequence Length=502026.06 | 100 | |
| ShellGamesSequence Length=2002026.06 | 99 | |
| ShellGamesSequence Length=1002026.06 | 98 | |
| ShellGamesSequence Length=Mean2026.06 | 97.6 | |
| ShellGamesSequence Length=502026.06 | 96 | |
| GPT-5.4-miniSequence Length=2002026.06 | 75 | |
| E-ShelLMSequence Length=502026.06 | 74 | |
| GPT-5.4-miniSequence Length=Mean2026.06 | 67.3 | |
| E-ShelLMSequence Length=1002026.06 | 60 | |
| E-ShelLMSequence Length=Mean2026.06 | 57 | |
| ShelLMSequence Length=502026.06 | 38 | |
| ShelLMSequence Length=2002026.06 | 37 | |
| E-ShelLMSequence Length=2002026.06 | 37 | |
| ShelLMSequence Length=Mean2026.06 | 33.6 | |
| GPT-5.4-miniSequence Length=1002026.06 | 27 | |
| ShelLMSequence Length=1002026.06 | 26 |