Reasoning on STB24
83.3Exact Match (EM)STAR (Claude Sonnet 4.6)
Evaluation results
| Method | Links | |
|---|---|---|
| STAR (Claude Sonnet 4.6)Framework=STAR, Params=–, Lat=14.0, Tok=5,5002026.05 | 83.3 | |
| STAR (Qwen3-8B)Framework=STAR, Params=8B, Lat=1.6, Tok=4,1912026.05 | 76.3 | |
| STARBackbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 76.3 | |
| STAR (Claude Haiku 4.5)Framework=STAR, Params=–, Lat=8.9, Tok=4,0732026.05 | 76.1 | |
| STAR (GPT-OSS-20B)Framework=STAR, Params=20B, Lat=7.1, Tok=5,3122026.05 | 71.8 | |
| STAR (Ministral-3-8B)Framework=STAR, Params=8B, Lat=1.9, Tok=4,6252026.05 | 66.8 | |
| GPT-OSS-20BFramework=LLM-only, Params=20B2026.05 | 51.4 | |
| STAR (GLM-4-9B)Framework=STAR, Params=9B, Lat=1.8, Tok=3,7462026.05 | 48.4 | |
| STAR (Llama-3.1-8B)Framework=STAR, Params=8B, Lat=2.2, Tok=4,1612026.05 | 48.2 | |
| Claude Sonnet 4.6Framework=LLM-only, Params=–2026.05 | 45.2 | |
| STAR (Llama-3.2-3B)Framework=STAR, Params=3B, Lat=1.3, Tok=4,0132026.05 | 43.1 | |
| ReflexionBackbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 37.2 | |
| GLM-4-9BFramework=LLM-only, Params=9B2026.05 | 37 | |
| LLM-only (CoT)Backbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 34.2 | |
| Ministral-3-8BFramework=LLM-only, Params=8B2026.05 | 32.1 | |
| Qwen3-8BFramework=LLM-only, Params=8B2026.05 | 31.9 | |
| LLM-onlyBackbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 31.9 | |
| ReActBackbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 31.8 | |
| Llama-3.1-8BFramework=LLM-only, Params=8B2026.05 | 31.5 | |
| Claude Haiku 4.5Framework=LLM-only, Params=–2026.05 | 28.7 | |
| Llama-3.2-3BFramework=LLM-only, Params=3B2026.05 | 22.3 | |
| Function-callingBackbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 20.8 | |
| Tree-of-ThoughtBackbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 11.8 | |
| Graph-of-ThoughtBackbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 10.2 |