List your company
ResearchFormerly WizWand

Question Answering on QA OOD StrQA SciQA

98.3StrQA AccuracyQwen3-8B pass@N (Upper Bound)
40.99655.87370.7585.627Nov 9, 2025
Updated 6mo ago

Evaluation results

MethodLinks
Qwen3-8B pass@N (Upper Bound)# Sample=-2025.1198.399.3——
Qwen3-14B pass@1# Sample=-2025.1191.995.6——
ReProbe, Attn+Logit, Self-anno# Sample=32K, Input Features=Attn+Logit, Annotation Source=Self-anno2025.1188.697.1——
ReProbe, Attn+Logit, DeepSeek-anno# Sample=32K, Input Features=Attn+Logit, Annotation Source=DeepSeek-anno2025.1188.696.9——
ReProbe, Hidden States, DeepSeek-anno# Sample=32K, Input Features=Hidden States, Annotation Source=DeepSeek-anno2025.1188.696.9——
Qwen2.5-Math-PRM-7B# Sample=860K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.1188.196.9——
ReProbe, Hidden States, Self-anno# Sample=32K, Input Features=Hidden States, Annotation Source=Self-anno2025.1188.195.8——
RLHFlow-PRM-Mistral-Data# Sample=273K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.1187.894.1——
Universal-PRM-Qwen2.5-Math-7B# Sample=690K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.1187.897.1——
RLHFlow-PRM-Deepseek-Data# Sample=253K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.1187.694.1——
Math-Shepherd-PRM-7B# Sample=440K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.1187.395.8——
Qwen2.5-Math-7B-PRM800k# Sample=263K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.1187.196.9——
Qwen3-8B pass@1 (Lower Bound)# Sample=-2025.1186.892.7——
Majority Voting# Sample=-2025.1186.692.5——
Skywork-PRM-1.5B# Sample=Unk, PRM Category=PRMs 150× Larger than ReProbes2025.1186.696.3——
H4-Qwen2.5-PRM-1.5B-0.2# Sample=369K, PRM Category=PRMs 150× Larger than ReProbes2025.1184.694.7——
Qwen3-1.7B pass@N (Upper Bound)# Sample=–, Base Model=Qwen3-1.7B, Decoding Strategy=pass@N, Inference Mode=native thinking mode2025.117489.281.683.4
Math-Shepherd-PRM-7B# Sample=440K, Base Model=Qwen3-1.7B, Decoding Strategy=Best-of-N=10, Inference Mode=native thinking mode2025.1154.647.250.962
ReProbe, Hidden States, GPT-OSS-anno# Sample=10.8K, Base Model=Qwen3-1.7B, Decoding Strategy=Best-of-N=10, Inference Mode=native thinking mode2025.1151.845.948.960.6
RLHFlow-PRM-Mistral-Data# Sample=273K, Base Model=Qwen3-1.7B, Decoding Strategy=Best-of-N=10, Inference Mode=native thinking mode2025.1151.249.850.559
MaxProb# Sample=–, Base Model=Qwen3-1.7B, Decoding Strategy=Best-of-N=10, Inference Mode=native thinking mode2025.1150.959.755.360.6
Perplexity# Sample=–, Base Model=Qwen3-1.7B, Decoding Strategy=Best-of-N=10, Inference Mode=native thinking mode2025.1150.959.755.360.6
RLHFlow-PRM-Deepseek-Data# Sample=253K, Base Model=Qwen3-1.7B, Decoding Strategy=Best-of-N=10, Inference Mode=native thinking mode2025.115045.647.858.6
MaxEntropy# Sample=–, Base Model=Qwen3-1.7B, Decoding Strategy=Best-of-N=10, Inference Mode=native thinking mode2025.1149.46054.760.5
Universal-PRM-Qwen2.5-Math-7B# Sample=690K, Base Model=Qwen3-1.7B, Decoding Strategy=Best-of-N=10, Inference Mode=native thinking mode2025.1148.557.45363
H4-Qwen2.5-PRM-1.5B-0.2# Sample=369K, Base Model=Qwen3-1.7B, Decoding Strategy=Best-of-N=10, Inference Mode=native thinking mode2025.1145.447.246.357.5
Qwen2.5-Math-PRM-7B# Sample=860K, Base Model=Qwen3-1.7B, Decoding Strategy=Best-of-N=10, Inference Mode=native thinking mode2025.1144.851.548.259.4
Qwen3-1.7B pass@1 (Lower Bound)# Sample=–, Base Model=Qwen3-1.7B, Decoding Strategy=pass@1, Inference Mode=native thinking mode2025.1143.248.345.852.8