Coding on Coding Tasks (test)
98.3Pass@1SALE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SALETask complexity (tau(t))=<= 0.12026.02 | 98.3 | 0.18 | |
| FrugalGPTTask complexity (tau(t))=<= 0.12026.02 | 97.5 | 0.39 | |
| SALE w/o memoryTask complexity (tau(t))=<= 0.12026.02 | 97.5 | 0.22 | |
| Best single agentTask complexity (tau(t))=<= 0.12026.02 | 95 | 0.36 | |
| CARROTTask complexity (tau(t))=<= 0.12026.02 | 95 | 0.36 | |
| TO-RouterTask complexity (tau(t))=<= 0.12026.02 | 95 | 0.36 | |
| WTPTask complexity (tau(t))=<= 0.12026.02 | 93.8 | 0.16 | |
| CARROTTask complexity (tau(t))=<= 0.52026.02 | 82.3 | 0.25 | |
| SALE w/o memoryTask complexity (tau(t))=<= 0.52026.02 | 82.3 | 0.28 | |
| SALETask complexity (tau(t))=<= 0.52026.02 | 82 | 0.27 | |
| Best single agentTask complexity (tau(t))=<= 0.52026.02 | 79.7 | 0.36 | |
| TO-RouterTask complexity (tau(t))=<= 0.52026.02 | 79.7 | 0.36 | |
| WTPTask complexity (tau(t))=<= 0.52026.02 | 76 | 0.15 | |
| FrugalGPTTask complexity (tau(t))=<= 0.52026.02 | 69.6 | 0.61 | |
| SALETask complexity (tau(t))=<= 2.52026.02 | 69 | 0.29 | |
| SALE w/o memoryTask complexity (tau(t))=<= 2.52026.02 | 68.8 | 0.31 | |
| Best single agentTask complexity (tau(t))=<= 2.52026.02 | 67.5 | 0.36 | |
| TO-RouterTask complexity (tau(t))=<= 2.52026.02 | 67.5 | 0.36 | |
| SALETask complexity (tau(t))=All2026.02 | 61.1 | 0.27 | |
| WTPTask complexity (tau(t))=<= 2.52026.02 | 60 | 0.15 | |
| CARROTTask complexity (tau(t))=<= 2.52026.02 | 60 | 0.26 | |
| SALE w/o memoryTask complexity (tau(t))=All2026.02 | 59.9 | 0.27 | |
| Best single agentTask complexity (tau(t))=All2026.02 | 58.4 | 0.36 | |
| TO-RouterTask complexity (tau(t))=All2026.02 | 58.4 | 0.36 | |
| CARROTTask complexity (tau(t))=All2026.02 | 57.1 | 0.31 | |
| FrugalGPTTask complexity (tau(t))=<= 2.52026.02 | 56.3 | 0.61 | |
| FrugalGPTTask complexity (tau(t))=All2026.02 | 50.4 | 0.57 | |
| WTPTask complexity (tau(t))=All2026.02 | 50.1 | 0.11 | |
| SALETask complexity (tau(t))=<= 12.52026.02 | 30.4 | 0.3 | |
| Best single agentTask complexity (tau(t))=<= 12.52026.02 | 27.2 | 0.36 | |
| CARROTTask complexity (tau(t))=<= 12.52026.02 | 27.2 | 0.36 | |
| TO-RouterTask complexity (tau(t))=<= 12.52026.02 | 27.2 | 0.36 | |
| SALE w/o memoryTask complexity (tau(t))=<= 12.52026.02 | 27.2 | 0.32 | |
| SALETask complexity (tau(t))=<= 602026.02 | 26.1 | 0.29 | |
| SALE w/o memoryTask complexity (tau(t))=<= 602026.02 | 24.1 | 0.31 | |
| Best single agentTask complexity (tau(t))=<= 602026.02 | 22.8 | 0.36 | |
| TO-RouterTask complexity (tau(t))=<= 602026.02 | 22.8 | 0.36 | |
| CARROTTask complexity (tau(t))=<= 602026.02 | 21.5 | 0.35 | |
| FrugalGPTTask complexity (tau(t))=<= 12.52026.02 | 18.5 | 0.61 | |
| WTPTask complexity (tau(t))=<= 12.52026.02 | 14.8 | 0.05 | |
| FrugalGPTTask complexity (tau(t))=<= 602026.02 | 10.1 | 0.61 | |
| WTPTask complexity (tau(t))=<= 602026.02 | 6.3 | 0.05 |