Question Answering on MMLU-Pro (Non-trivial subset All 14 subjects, test)
33.93Accuracyoracle (pass@16)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| oracle (pass@16)Solver=Qwen3-1.7B, Temperature=0.7, Samples per question=128, Embeddings=Qwen3-Embedding-8B, Number of voters=162026.06 | 33.93 | 5.93 | |
| PPV confidenceSolver=Qwen3-1.7B, Temperature=0.7, Samples per question=128, Embeddings=Qwen3-Embedding-8B, Number of voters=16, Mode=confidence2026.06 | 30.24 | 2.24 | |
| PPV divSolver=Qwen3-1.7B, Temperature=0.7, Samples per question=128, Embeddings=Qwen3-Embedding-8B, Number of voters=16, Mode=div2026.06 | 29.83 | 1.83 | |
| PPV inverted x divSolver=Qwen3-1.7B, Temperature=0.7, Samples per question=128, Embeddings=Qwen3-Embedding-8B, Number of voters=16, Mode=inverted x div2026.06 | 29.52 | 1.52 | |
| majoritySolver=Qwen3-1.7B, Temperature=0.7, Samples per question=128, Embeddings=Qwen3-Embedding-8B, Number of voters=162026.06 | 28 | — |