Reasoning on Brumo 25
613Trace CountUniform
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| UniformLLM=Qwen3-4B, Confidence-Weighted=false2026.02 | 613 | 94 | 61.5 | |
| PETS-On.LLM=Qwen3-4B, Confidence-Weighted=false2026.02 | 601 | 100 | 63.2 | |
| UniformLLM=GPT-120B, Confidence-Weighted=false2026.02 | 521 | 92 | 86.3 | |
| Uniform (conf)LLM=Qwen3-4B, Confidence-Weighted=true2026.02 | 519 | 93 | 65.5 | |
| PETS-On. (conf)LLM=Qwen3-4B, Confidence-Weighted=true2026.02 | 494 | 97 | 64.7 | |
| PETS-On.LLM=GPT-120B, Confidence-Weighted=false2026.02 | 383 | 100 | 88.8 | |
| Uniform (conf)LLM=GPT-120B, Confidence-Weighted=true2026.02 | 353 | 94 | 87.2 | |
| UniformLLM=GPT-20B, Confidence-Weighted=false2026.02 | 245 | 95 | 91.7 | |
| PETS-On. (conf)LLM=GPT-120B, Confidence-Weighted=true2026.02 | 241 | 99 | 89.7 | |
| PETS-On.LLM=GPT-20B, Confidence-Weighted=false2026.02 | 213 | 100 | 95 | |
| Uniform (conf)LLM=Qwen-Long, Confidence-Weighted=true2026.02 | 189 | 98 | 81.3 | |
| Uniform (conf)LLM=GPT-20B, Confidence-Weighted=true2026.02 | 184 | 95 | 91.7 | |
| UniformLLM=Qwen3-30B, Confidence-Weighted=false2026.02 | 172 | 96 | 77.7 | |
| UniformLLM=Qwen-Long, Confidence-Weighted=false2026.02 | 167 | 97 | 80.7 | |
| PETS-On. (conf)LLM=GPT-20B, Confidence-Weighted=true2026.02 | 162 | 99 | 94.7 | |
| Uniform (conf)LLM=Qwen3-30B, Confidence-Weighted=true2026.02 | 159 | 96 | 77.7 | |
| PETS-On. (conf)LLM=Qwen-Long, Confidence-Weighted=true2026.02 | 138 | 99 | 81.5 | |
| PETS-On.LLM=Qwen-Long, Confidence-Weighted=false2026.02 | 130 | 100 | 80.2 | |
| PETS-On.LLM=Qwen3-30B, Confidence-Weighted=false2026.02 | 125 | 100 | 80 | |
| PETS-On. (conf)LLM=Qwen3-30B, Confidence-Weighted=true2026.02 | 121 | 99 | 79.5 |