Reasoning on STARK
79.3EMSTAR (Claude Sonnet 4.6)
Evaluation results
| Method | Links | |
|---|---|---|
| STAR (Claude Sonnet 4.6)Framework=STAR, Params=–, Lat=14.0, Tok=5,5002026.05 | 79.3 | |
| STAR (GPT-OSS-20B)Framework=STAR, Params=20B, Lat=7.1, Tok=5,3122026.05 | 75.1 | |
| STAR (Claude Haiku 4.5)Framework=STAR, Params=–, Lat=8.9, Tok=4,0732026.05 | 74.6 | |
| STAR (Qwen3-8B)Framework=STAR, Params=8B, Lat=1.6, Tok=4,1912026.05 | 73 | |
| STARBackbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 73 | |
| Claude Sonnet 4.6Framework=LLM-only, Params=–2026.05 | 71.4 | |
| Claude Haiku 4.5Framework=LLM-only, Params=–2026.05 | 70.3 | |
| ReflexionBackbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 68.8 | |
| GPT-OSS-20BFramework=LLM-only, Params=20B2026.05 | 67.9 | |
| STAR (Llama-3.1-8B)Framework=STAR, Params=8B, Lat=2.2, Tok=4,1612026.05 | 67.8 | |
| ReActBackbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 65.6 | |
| STAR (GLM-4-9B)Framework=STAR, Params=9B, Lat=1.8, Tok=3,7462026.05 | 65.3 | |
| STAR (Ministral-3-8B)Framework=STAR, Params=8B, Lat=1.9, Tok=4,6252026.05 | 64.1 | |
| Tree-of-ThoughtBackbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 62.8 | |
| Graph-of-ThoughtBackbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 61.7 | |
| LLM-only (CoT)Backbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 59.4 | |
| Qwen3-8BFramework=LLM-only, Params=8B2026.05 | 55.4 | |
| LLM-onlyBackbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 55.4 | |
| GLM-4-9BFramework=LLM-only, Params=9B2026.05 | 52.9 | |
| STAR (Llama-3.2-3B)Framework=STAR, Params=3B, Lat=1.3, Tok=4,0132026.05 | 51.8 | |
| Ministral-3-8BFramework=LLM-only, Params=8B2026.05 | 51.3 | |
| Llama-3.1-8BFramework=LLM-only, Params=8B2026.05 | 50.4 | |
| Llama-3.2-3BFramework=LLM-only, Params=3B2026.05 | 41.8 | |
| Function-callingBackbone=Qwen3-8B, Query Eligibility=EM-eligible queries2026.05 | 31.9 |