Interactive Decision Making on InterCode NL2Bash
79.6Success RateGRASP
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| GRASPModel=DeepSeek V3.22026.04 | 79.6 | 11.2 | |
| ReAct + SkillsModel=DeepSeek V3.22026.04 | 79.1 | 13.2 | |
| GRASPModel=Gemini 2.5 Pro2026.04 | 78.3 | 12 | |
| ReAct + SkillsModel=Gemini 2.5 Pro2026.04 | 77.7 | 14.1 | |
| GRASPModel=Claude-4-Sonnet2026.04 | 77.6 | 13.2 | |
| GRASPModel=Kimi-K2.52026.04 | 77.4 | 12.8 | |
| ReAct + SkillsModel=Kimi-K2.52026.04 | 76.8 | 13.6 | |
| GRASPModel=o4 Mini2026.04 | 76.7 | 13.8 | |
| ReAct + SkillsModel=Claude-4-Sonnet2026.04 | 76.7 | 13.2 | |
| GRASPModel=GLM-52026.04 | 76.7 | 13.4 | |
| ReflexionModel=DeepSeek V3.22026.04 | 75.8 | 14 | |
| GRASPModel=GPT-4.12026.04 | 75.8 | 12.5 | |
| ReAct + SkillsModel=GLM-52026.04 | 75.6 | 14.6 | |
| ReAct + SkillsModel=GPT-4.12026.04 | 75.1 | 13.3 | |
| ReflexionModel=Gemini 2.5 Pro2026.04 | 74.3 | 14.6 | |
| ExpeLModel=DeepSeek V3.22026.04 | 74.2 | 14.3 | |
| GRASPModel=Qwen3-235B2026.04 | 74.2 | 13.4 | |
| ReflexionModel=Claude-4-Sonnet2026.04 | 73.6 | 14 | |
| ReAct + SkillsModel=Qwen3-235B2026.04 | 73.5 | 13.8 | |
| ExpeLModel=Gemini 2.5 Pro2026.04 | 73.4 | 14.8 | |
| ExpeLModel=Claude-4-Sonnet2026.04 | 73.4 | 13.6 | |
| ReflexionModel=Kimi-K2.52026.04 | 73.4 | 14.2 | |
| ReActModel=Kimi-K2.52026.04 | 73.2 | 15 | |
| ExpeLModel=Kimi-K2.52026.04 | 73.2 | 14.6 | |
| ReActModel=DeepSeek V3.22026.04 | 72.6 | 14.8 | |
| ReActModel=GPT-4.12026.04 | 72.4 | 14.6 | |
| ReflexionModel=GPT-4.12026.04 | 72.4 | 13.8 | |
| ReflexionModel=GLM-52026.04 | 72.1 | 15.2 | |
| ReAct + SkillsModel=o4 Mini2026.04 | 71.6 | 15.1 | |
| ReActModel=Qwen3-235B2026.04 | 71.6 | 15.4 | |
| ReActModel=Gemini 2.5 Pro2026.04 | 70.8 | 15.6 | |
| ExpeLModel=GPT-4.12026.04 | 70.8 | 14.6 | |
| ReflexionModel=Qwen3-235B2026.04 | 70.8 | 14.6 | |
| ReActModel=Claude-4-Sonnet2026.04 | 70.8 | 14.8 | |
| ExpeLModel=Qwen3-235B2026.04 | 70.6 | 14.2 | |
| ReflexionModel=o4 Mini2026.04 | 70.2 | 15.8 | |
| ReActModel=GLM-52026.04 | 68.4 | 16.5 | |
| ExpeLModel=o4 Mini2026.04 | 67.9 | 16.2 | |
| ExpeLModel=GLM-52026.04 | 67.9 | 16 | |
| ReActModel=o4 Mini2026.04 | 66.4 | 17.2 |