Reasoning on STB26
64.5Exact Match (EM)STAR (Claude Sonnet 4.6)
Evaluation results
| Method | Links | |
|---|---|---|
| STAR (Claude Sonnet 4.6)Framework=STAR, Params=–, Lat=14.0, Tok=5,5002026.05 | 64.5 | |
| STAR (GPT-OSS-20B)Framework=STAR, Params=20B, Lat=7.1, Tok=5,3122026.05 | 59.2 | |
| STAR (Qwen3-8B)Framework=STAR, Params=8B, Lat=1.6, Tok=4,1912026.05 | 55.3 | |
| STAR (Claude Haiku 4.5)Framework=STAR, Params=–, Lat=8.9, Tok=4,0732026.05 | 55 | |
| STAR (GLM-4-9B)Framework=STAR, Params=9B, Lat=1.8, Tok=3,7462026.05 | 48.3 | |
| GPT-OSS-20BFramework=LLM-only, Params=20B2026.05 | 47.2 | |
| STAR (Ministral-3-8B)Framework=STAR, Params=8B, Lat=1.9, Tok=4,6252026.05 | 46.2 | |
| Claude Haiku 4.5Framework=LLM-only, Params=–2026.05 | 45.6 | |
| GLM-4-9BFramework=LLM-only, Params=9B2026.05 | 45.6 | |
| STAR (Llama-3.1-8B)Framework=STAR, Params=8B, Lat=2.2, Tok=4,1612026.05 | 43.9 | |
| Ministral-3-8BFramework=LLM-only, Params=8B2026.05 | 42 | |
| STAR (Llama-3.2-3B)Framework=STAR, Params=3B, Lat=1.3, Tok=4,0132026.05 | 39.1 | |
| Llama-3.1-8BFramework=LLM-only, Params=8B2026.05 | 38.6 | |
| Qwen3-8BFramework=LLM-only, Params=8B2026.05 | 37.5 | |
| Llama-3.2-3BFramework=LLM-only, Params=3B2026.05 | 32 | |
| Claude Sonnet 4.6Framework=LLM-only, Params=–2026.05 | 30.1 |