Bash Command Execution on InterCode Bash
72Execution Success RateGPT-5
Evaluation Results
| Method | Links | |
|---|---|---|
| GPT-5Method=Few-Shot, Avg Latency (ms)=∼16,4902026.04 | 72 | |
| Meta-Tool (3B)Method=Ours, Avg Latency (ms)=∼1,5762026.04 | 54 | |
| QWEN3.5-397B-A17BBackbone LLM=QWEN3.5-397B-A17B2026.06 | 51.34 | |
| OPD-Evolver-9BBackbone LLM=QWEN3.5-9B2026.06 | 49.55 | |
| MemEvolveBackbone LLM=QWEN3.5-9B2026.06 | 45.98 | |
| EvolveRBackbone LLM=QWEN3.5-9B2026.06 | 44.64 | |
| STEP-3.5-FLASH (196B)Backbone LLM=STEP-3.5-FLASH (196B)2026.06 | 44.2 | |
| MempBackbone LLM=QWEN3.5-9B2026.06 | 44.2 | |
| Llama-3.2-3B-Ins.Method=Few-Shot, Avg Latency (ms)=∼1,6212026.04 | 44 | |
| ReasoningBankBackbone LLM=QWEN3.5-9B2026.06 | 43.75 | |
| CheatsheetBackbone LLM=QWEN3.5-9B2026.06 | 41.96 | |
| No MemoryBackbone LLM=QWEN3.5-9B2026.06 | 41.52 | |
| ExpeLBackbone LLM=QWEN3.5-9B2026.06 | 40.18 | |
| AgentLM-7BMethod=Few-Shot, Avg Latency (ms)=∼8,8802026.04 | 40 | |
| AWMBackbone LLM=QWEN3.5-9B2026.06 | 39.73 | |
| OPD-Evolver-4BBackbone LLM=QWEN3-4B2026.06 | 36.16 | |
| EvolveRBackbone LLM=QWEN3-4B2026.06 | 33.93 | |
| MemEvolveBackbone LLM=QWEN3-4B2026.06 | 33.93 | |
| CheatsheetBackbone LLM=QWEN3-4B2026.06 | 33.04 | |
| MempBackbone LLM=QWEN3-4B2026.06 | 31.7 | |
| ReasoningBankBackbone LLM=QWEN3-4B2026.06 | 31.25 | |
| ExpeLBackbone LLM=QWEN3-4B2026.06 | 30.8 | |
| No MemoryBackbone LLM=QWEN3-4B2026.06 | 30.36 | |
| AWMBackbone LLM=QWEN3-4B2026.06 | 27.68 |