Reasoning on HMMT Feb 25
100ConsistencyPETS-On.
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| PETS-On.LLM=Qwen3-4B, Confidence-Weighted=false2026.02 | 100 | 531 | 48.5 | |
| PETS-On.LLM=Qwen3-30B, Confidence-Weighted=false2026.02 | 100 | 825 | 56.2 | |
| PETS-On.LLM=Qwen-Long, Confidence-Weighted=false2026.02 | 100 | 872 | 67 | |
| PETS-On.LLM=GPT-20B, Confidence-Weighted=false2026.02 | 100 | 248 | 85 | |
| PETS-On.LLM=GPT-120B, Confidence-Weighted=false2026.02 | 100 | 221 | 85 | |
| PETS-On. (conf)LLM=GPT-20B, Confidence-Weighted=true2026.02 | 97 | 233 | 85.5 | |
| UniformLLM=Qwen-Long, Confidence-Weighted=false2026.02 | 96 | — | 66.7 | |
| PETS-On. (conf)LLM=GPT-120B, Confidence-Weighted=true2026.02 | 96 | 263 | 86.5 | |
| PETS-On. (conf)LLM=Qwen3-30B, Confidence-Weighted=true2026.02 | 95 | 761 | 56.8 | |
| PETS-On. (conf)LLM=Qwen-Long, Confidence-Weighted=true2026.02 | 95 | 1,061 | 68.8 | |
| Uniform (conf)LLM=Qwen-Long, Confidence-Weighted=true2026.02 | 95 | 959 | 67.8 | |
| UniformLLM=GPT-120B, Confidence-Weighted=false2026.02 | 95 | 309 | 84.8 | |
| PETS-On. (conf)LLM=Qwen3-4B, Confidence-Weighted=true2026.02 | 93 | 615 | 46.8 | |
| UniformLLM=Qwen3-30B, Confidence-Weighted=false2026.02 | 93 | 813 | 56.2 | |
| Uniform (conf)LLM=Qwen3-30B, Confidence-Weighted=true2026.02 | 92 | 786 | 56.7 | |
| UniformLLM=GPT-20B, Confidence-Weighted=false2026.02 | 92 | 475 | 83.3 | |
| Uniform (conf)LLM=GPT-20B, Confidence-Weighted=true2026.02 | 92 | 409 | 83.2 | |
| Uniform (conf)LLM=GPT-120B, Confidence-Weighted=true2026.02 | 91 | 458 | 85 | |
| UniformLLM=Qwen3-4B, Confidence-Weighted=false2026.02 | 89 | 792 | 46.2 | |
| Uniform (conf)LLM=Qwen3-4B, Confidence-Weighted=true2026.02 | 89 | 931 | 45.7 |