Macro-average Exact Match on ShellOps-Pro
53.3Macro-average Exact Match AccuracyKimi-K2.6
Evaluation Results
| Method | Links | |
|---|---|---|
| Kimi-K2.6Hmax=10, Paradigm=Frontier Agentic Inference2026.05 | 53.3 | |
| Kimi-K2.6Hmax=8, Paradigm=Frontier Agentic Inference2026.05 | 47.9 | |
| Kimi-K2.6Hmax=6, Paradigm=Frontier Agentic Inference2026.05 | 45.9 | |
| GLM-5.1Hmax=10, Paradigm=Frontier Agentic Inference2026.05 | 40.7 | |
| GLM-5.1Hmax=8, Paradigm=Frontier Agentic Inference2026.05 | 39.1 | |
| GLM-5.1Hmax=6, Paradigm=Frontier Agentic Inference2026.05 | 34.7 | |
| A3 (σ-Reveal)Hmax=10, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 31.1 | |
| A3 (σ-Reveal)Hmax=8, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 30.5 | |
| A3 (σ-Reveal)Hmax=6, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 29.9 | |
| Qwen3-235B-A22BHmax=8, Paradigm=Frontier Agentic Inference2026.05 | 29.3 | |
| Qwen3-235B-A22BHmax=10, Paradigm=Frontier Agentic Inference2026.05 | 28.3 | |
| GiGPOHmax=6, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 27.1 | |
| Qwen3-235B-A22BHmax=6, Paradigm=Frontier Agentic Inference2026.05 | 26.7 | |
| GiGPOHmax=8, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 26.3 | |
| A3 (Vanilla)Hmax=8, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 25.9 | |
| GiGPOHmax=10, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 25.7 | |
| GSPOHmax=10, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 25.5 | |
| A3 (Vanilla)Hmax=10, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 25.3 | |
| GSPOHmax=8, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 24.9 | |
| GSPOHmax=6, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 24.4 | |
| A3 (Vanilla)Hmax=6, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 22.9 | |
| HGPOHmax=10, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 22.8 | |
| ReACTHmax=8, Paradigm=Agentic Inference, Backbone=Qwen3-14B2026.05 | 22.3 | |
| ReACTHmax=10, Paradigm=Agentic Inference, Backbone=Qwen3-14B2026.05 | 22 | |
| HGPOHmax=8, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 21.9 | |
| LATSHmax=8, Paradigm=Agentic Inference, Backbone=Qwen3-14B2026.05 | 21.7 | |
| LATSHmax=6, Paradigm=Agentic Inference, Backbone=Qwen3-14B2026.05 | 21.1 | |
| rStarHmax=8, Paradigm=Agentic Inference, Backbone=Qwen3-14B2026.05 | 21.1 | |
| rStarHmax=10, Paradigm=Agentic Inference, Backbone=Qwen3-14B2026.05 | 20.4 | |
| RetroAgentHmax=10, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 20 | |
| LATSHmax=10, Paradigm=Agentic Inference, Backbone=Qwen3-14B2026.05 | 19.7 | |
| HGPOHmax=6, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 19.5 | |
| ReACTHmax=6, Paradigm=Agentic Inference, Backbone=Qwen3-14B2026.05 | 19.3 | |
| rStarHmax=6, Paradigm=Agentic Inference, Backbone=Qwen3-14B2026.05 | 19.3 | |
| RetroAgentHmax=8, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 19.2 | |
| RetroAgentHmax=6, Paradigm=Agentic RL, Backbone=Qwen3-14B2026.05 | 18 |