Software Engineering Reasoning on CLAW-Eval
61.5Pass@3TACT-GATE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| TACT-GATEBackbone=Qwen3.5-27B2026.05 | 61.5 | 10.8 | |
| TACT-GATEBackbone=Gemma-4-26B-A4B-it2026.05 | 61.4 | 8.1 | |
| TACT-ADDBackbone=Qwen3.5-27B2026.05 | 61.2 | 11.1 | |
| TACT-ADDBackbone=Gemma-4-26B-A4B-it2026.05 | 61 | 8.4 | |
| AGENTDEBUGBackbone=Qwen3.5-27B, rollouts=at least two per task2026.05 | 58.2 | 16.9 | |
| TACT-CAPBackbone=Gemma-4-26B-A4B-it2026.05 | 58.2 | 7 | |
| AGENTDEBUGBackbone=Gemma-4-26B-A4B-it, rollouts=at least two per task2026.05 | 58 | 12.6 | |
| TACT-CAPBackbone=Qwen3.5-27B2026.05 | 57.8 | 9.4 | |
| RECAPBackbone=Gemma-4-26B-A4B-it2026.05 | 57.6 | 9.1 | |
| AGENTREFLECTBackbone=Qwen3.5-27B2026.05 | 57 | 11.9 | |
| AGENTREFLECTBackbone=Gemma-4-26B-A4B-it2026.05 | 56.9 | 9 | |
| REMINDERBackbone=Gemma-4-26B-A4B-it2026.05 | 56.8 | 9.3 | |
| RECAPBackbone=Qwen3.5-27B2026.05 | 56.4 | 12.5 | |
| Base ModelBackbone=Gemma-4-26B-A4B-it2026.05 | 56.3 | 9.2 | |
| Base ModelBackbone=Qwen3.5-27B2026.05 | 55.9 | 12.3 | |
| REMINDERBackbone=Qwen3.5-27B2026.05 | 55.5 | 12.7 |