Software Engineering on SWE-bench Lite first 10 instances (test)
0FCRgroq:llama-3.3-70b
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| groq:llama-3.3-70bProtocol=Unwrapped, Wall-clock cap=15-minute2026.07 | 0 | 10 | — | — | — | |
| groq:llama-3.3-70bProtocol=HCRC, Wall-clock cap=15-minute2026.07 | 0 | — | 7 | 0 | 6.2 | |
| openrouter:gpt-oss-120BProtocol=Unwrapped, Wall-clock cap=15-minute2026.07 | 0 | 6 | — | — | — | |
| openrouter:gpt-oss-120BProtocol=HCRC, Wall-clock cap=15-minute2026.07 | 0 | — | 8 | 0 | 58 | |
| groq:llama-3.1-8bProtocol=Unwrapped, Wall-clock cap=15-minute2026.07 | 20 | 10 | — | — | — | |
| groq:llama-3.1-8bProtocol=HCRC, Wall-clock cap=15-minute2026.07 | 20 | — | 10 | 1.5 | 34.2 | |
| gemini-2.5-flashProtocol=Unwrapped, Wall-clock cap=15-minute2026.07 | — | 4 | — | — | — | |
| gemini-2.5-flashProtocol=HCRC, Wall-clock cap=15-minute2026.07 | — | — | 0 | — | — |