Code Generation on TerminalBench 2 (Pass@3)
39.3Pass@3MTL (I)
Evaluation results
| Method | Links | |
|---|---|---|
| MTL (I)Model=DeepSeek V3.2, Method=MTL (I)2026.04 | 39.3 | |
| MTL (S)Model=GPT-5-mini, Method=MTL (S)2026.04 | 37.1 | |
| MTL (I)Model=GPT-5-mini, Method=MTL (I)2026.04 | 36 | |
| MTL (W)Model=GPT-5-mini, Method=MTL (W)2026.04 | 34.8 | |
| Zero-shotModel=DeepSeek V3.2, Method=Zero-shot2026.04 | 33.7 | |
| MTL (I)Model=Qwen3-Coder-480B-A35B-Instruct, Method=MTL (I)2026.04 | 32.6 | |
| Zero-shotModel=GPT-5-mini, Method=Zero-shot2026.04 | 31.5 | |
| Zero-shotModel=Qwen3-Coder-480B-A35B-Instruct, Method=Zero-shot2026.04 | 29.2 | |
| MTL (T)Model=GPT-5-mini, Method=MTL (T)2026.04 | 27 |