Question Answering on MMLU-PRO (test) Subject Scores and Pass@1
56.37Pass@1 AccuracyRLVR
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| RLVRBackbone=QWEN2.5-7B-INSTRUCT, Training Dataset=WEBINSTRUCT-VERIFIED2025.10 | 56.37 | 72.11 | 62.96 | 51.86 | 57.34 | 42.39 | |
| RLERBackbone=QWEN2.5-7B-INSTRUCT, Training Dataset=WEBINSTRUCT-VERIFIED, k=2, Gk=82025.10 | 55.12 | 70.83 | 61.59 | 52.22 | 56.28 | 41.3 | |
| Co-rewardingBackbone=QWEN2.5-7B-INSTRUCT, Training Dataset=WEBINSTRUCT-VERIFIED, Training Mode=RLIR2025.10 | 54.82 | 70.25 | 61.45 | 51.87 | 56.13 | 40.27 | |
| SCBackbone=QWEN2.5-7B-INSTRUCT, Training Dataset=WEBINSTRUCT-VERIFIED, Training Mode=RLIR2025.10 | 54.05 | 70.29 | 60.71 | 53.27 | 56.1 | 40.56 | |
| INTUITORBackbone=QWEN2.5-7B-INSTRUCT, Training Dataset=WEBINSTRUCT-VERIFIED, Training Mode=RLIR2025.10 | 53.06 | 70.05 | 61.23 | 51.11 | 56.32 | 39.87 | |
| JudgeBackbone=QWEN2.5-7B-INSTRUCT, Training Dataset=WEBINSTRUCT-VERIFIED, Training Mode=RLIR2025.10 | 52.82 | 70.71 | 61.72 | 49.56 | 57.56 | 37.46 | |
| pre-RLBackbone=QWEN2.5-7B-INSTRUCT, Training Dataset=WEBINSTRUCT-VERIFIED2025.10 | 48.72 | 59.55 | 59.7 | 48.23 | 52.44 | 34.47 | |
| oracle (pass@16)Solver=Qwen3-1.7B, Temperature=0.7, Samples per question=128, Embeddings=Qwen3-Embedding-8B, Number of voters=162026.06 | 44.68 | — | — | — | — | — | |
| PPV confidenceSolver=Qwen3-1.7B, Temperature=0.7, Samples per question=128, Embeddings=Qwen3-Embedding-8B, Number of voters=16, Mode=confidence2026.06 | 42.21 | — | — | — | — | — | |
| PPV divSolver=Qwen3-1.7B, Temperature=0.7, Samples per question=128, Embeddings=Qwen3-Embedding-8B, Number of voters=16, Mode=div2026.06 | 41.92 | — | — | — | — | — | |
| PPV inverted x divSolver=Qwen3-1.7B, Temperature=0.7, Samples per question=128, Embeddings=Qwen3-Embedding-8B, Number of voters=16, Mode=inverted x div2026.06 | 41.71 | — | — | — | — | — | |
| majoritySolver=Qwen3-1.7B, Temperature=0.7, Samples per question=128, Embeddings=Qwen3-Embedding-8B, Number of voters=162026.06 | 40.71 | — | — | — | — | — | |
| best dictatorSolver=Qwen3-1.7B, Temperature=0.7, Samples per question=128, Embeddings=Qwen3-Embedding-8B, Number of voters=162026.06 | 35.78 | — | — | — | — | — |