Scientific Reasoning on TheoremQA
82.49AccuracyUpper Bound
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Upper BoundBackbone=Gemma-2-9B-Instruct, k=322025.08 | 82.49 | — | |
| Upper BoundBackbone=Gemma-2-9B-Instruct, k=62025.08 | 55.22 | — | |
| Upper BoundModel=DS-Distill-Qwen-2.5-7B, k=62025.08 | 48.59 | — | |
| Upper BoundModel=Qwen3-8B, k=62025.08 | 44.71 | — | |
| Upper BoundBackbone=Llama-3.1-70B-Instruct, k=162025.08 | 43.47 | — | |
| Upper BoundBackbone=Qwen3-32B (Non-thinking), k=162025.08 | 42.93 | — | |
| PiCSARModel=Qwen3-8B, k=62025.08 | 42.57 | — | |
| TABBudget (B)=10k2026.04 | 42.3 | 6,566 | |
| StaticBudget=40962026.04 | 42.1 | 10,754 | |
| Self-ConsistencyModel=Qwen3-8B, k=62025.08 | 41.81 | — | |
| TABBudget (B)=8k2026.04 | 41.5 | 4,987 | |
| CISC (PiCSAR)Backbone=Llama-3.1-8B-Instruct, k=322025.08 | 41.5 | — | |
| TABBudget (B)=5k2026.04 | 40.9 | 3,141 | |
| AverageModel=Qwen3-8B, k=62025.08 | 40.83 | — | |
| LLM-Judge Multi-TurnSelection Strategy=Multi-Turn2026.04 | 40.8 | 5,375 | |
| Upper BoundBackbone=Llama-3.1-70B-Instruct, k=62025.08 | 40.7 | — | |
| StaticBudget=20482026.04 | 40.3 | 5,988 | |
| Upper BoundBackbone=Qwen3-8B (Non-thinking), k=322025.08 | 40.03 | — | |
| StaticBudget=10242026.04 | 40 | 3,975 | |
| TABBudget (B)=3k2026.04 | 39.9 | 2,320 | |
| LLM-Judge IndividualSelection Strategy=Individual2026.04 | 39.8 | 5,916 | |
| Upper BoundBackbone=Qwen3-32B (Non-thinking), k=62025.08 | 39.76 | — | |
| StaticBudget=5122026.04 | 39.5 | 2,578 | |
| CISC (p(True))Backbone=Llama-3.1-8B-Instruct, k=322025.08 | 39.38 | — | |
| StaticBudget=2562026.04 | 38.1 | 1,778 | |
| Upper BoundBackbone=Llama-3.1-8B-Instruct, k=322025.08 | 37.846 | — | |
| PiCSARModel=DS-Distill-Qwen-2.5-7B, k=62025.08 | 36.76 | — | |
| CICS (PiCSAR)Backbone=Qwen3-8B (Non-thinking), k=62025.08 | 36.46 | — | |
| CICS (PiCSAR)Backbone=Qwen3-32B (Non-thinking), k=62025.08 | 36.46 | — | |
| Upper BoundModel=DS-Distill-llama-3-8B, k=62025.08 | 36.37 | — | |
| CICS (PiCSAR)Backbone=Qwen3-8B (Non-thinking), k=322025.08 | 36.32 | — | |
| CICS (PiCSAR)Backbone=Qwen3-32B (Non-thinking), k=162025.08 | 36.32 | — | |
| Self-ConsistencyModel=DS-Distill-Qwen-2.5-7B, k=62025.08 | 35.88 | — | |
| Upper BoundBackbone=Qwen3-8B (Non-thinking), k=62025.08 | 34.94 | — | |
| AverageModel=DS-Distill-Qwen-2.5-7B, k=62025.08 | 33.11 | — | |
| CICS (p(True))Backbone=Qwen3-8B (Non-thinking), k=322025.08 | 32.75 | — | |
| CICS (P-True)Backbone=Qwen3-32B (Non-thinking), k=162025.08 | 32.75 | — | |
| CISC (PiCSAR)Backbone=Llama-3.1-70B-Instruct, k=62025.08 | 31.91 | — | |
| CISC (PiCSAR)Backbone=Llama-3.1-70B-Instruct, k=162025.08 | 31.59 | — | |
| PiCSARBackbone=Qwen3-32B (Non-thinking), k=62025.08 | 31.46 | — | |
| PiCSARModel=Qwen3-32B (Non-thinking), k=62025.08 | 31.46 | — | |
| PiCSARBackbone=Qwen3-32B (Non-thinking), k=162025.08 | 31.42 | — | |
| CICS (p(True))Backbone=Qwen3-8B (Non-thinking), k=62025.08 | 31.24 | — | |
| CICS (P-True)Backbone=Qwen3-32B (Non-thinking), k=62025.08 | 31.24 | — | |
| PiCSARBackbone=Qwen3-8B (Non-thinking), k=62025.08 | 29.76 | — | |
| PiCSARModel=Qwen3-8B (Non-thinking), k=62025.08 | 29.76 | — | |
| CISC (p(True))Backbone=Llama-3.1-70B-Instruct, k=162025.08 | 29.41 | — | |
| PiCSARBackbone=Qwen3-8B (Non-thinking), k=322025.08 | 29.17 | — | |
| Upper BoundBackbone=Llama-3.1-8B-Instruct, k=62025.08 | 28.2 | — | |
| CISC (p(True))Backbone=Llama-3.1-70B-Instruct, k=62025.08 | 28.07 | — | |
| PiCSARModel=DS-Distill-llama-3-8B, k=62025.08 | 28.02 | — | |
| Self-ConsistencyModel=DS-Distill-llama-3-8B, k=62025.08 | 27.98 | — | |
| PiCSARBackbone=Llama-3.1-70B-Instruct, k=62025.08 | 27.84 | — | |
| PiCSARModel=Llama-3.1-70B-Instruct, k=62025.08 | 27.84 | — | |
| PiCSARBackbone=Llama-3.1-70B-Instruct, k=162025.08 | 26.73 | — | |
| AverageModel=DS-Distill-llama-3-8B, k=62025.08 | 26.58 | — | |
| CISC (PiCSAR)Backbone=Gemma-2-9B-Instruct, k=322025.08 | 23.16 | — | |
| CISC (PiCSAR)Backbone=Llama-3.1-8B-Instruct, k=62025.08 | 22.71 | — | |
| CISC (PiCSAR)Backbone=Gemma-2-9B-Instruct, k=62025.08 | 21.02 | — | |
| PiCSARBackbone=Llama-3.1-8B-Instruct, k=62025.08 | 20.08 | — | |
| PiCSARModel=Llama-3.1-8B-Instruct, k=62025.08 | 20.08 | — | |
| PiCSARBackbone=Llama-3.1-8B-Instruct, k=322025.08 | 19.72 | — | |
| PiCSARBackbone=Gemma-2-9B-Instruct, k=322025.08 | 18.88 | — | |
| PiCSARBackbone=Gemma-2-9B-Instruct, k=62025.08 | 18.62 | — | |
| PiCSARModel=Gemma-2-9B-Instruct, k=62025.08 | 18.62 | — | |
| CISC (p(True))Backbone=Llama-3.1-8B-Instruct, k=62025.08 | 18.03 | — | |
| CISC (p(True))Backbone=Gemma-2-9B-Instruct, k=322025.08 | 17.45 | — | |
| CISC (p(True))Backbone=Gemma-2-9B-Instruct, k=62025.08 | 17.09 | — |