Task Success and Utility Evaluation on Suite-C
100Success RateFull access
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Full accessCondition=Full access, Base Model=Qwen3-Coder 30B2026.06 | 100 | 69 | 31 | |
| Static allowlistCondition=Static allowlist, Base Model=Qwen3-Coder 30B2026.06 | 100 | 70 | 30 | |
| Coarse sandboxCondition=Coarse sandbox, Base Model=Qwen3-Coder 30B2026.06 | 100 | 70 | 30 | |
| PORTICO fullCondition=PORTICO full, Base Model=Qwen3-Coder 30B2026.06 | 100 | 100 | 0 | |
| PORTICO coreCondition=PORTICO core, Base Model=Qwen3-Coder 30B2026.06 | 94 | 100 | 0 |