List your company
ResearchFormerly WizWand

Uncertainty Estimation on APPS Intro

0.813AUROCTop-5 Entropy (LEX)
0.435480.533490.63150.72951Jun 8, 2026
Updated 4mo ago

Evaluation results

MethodLinks
Top-5 Entropy (LEX)Model=Qwen3-14B2026.060.8130.828
Top-5 Entropy (LEX)Backbone=Qwen3-14B2026.060.8130.828
Top-5 Entropy (LEX)Model=Qwen3-Coder-30B2026.060.8070.822
Pseudo Consistency (ALGO)Model=Devstral-Small-25052026.060.8030.704
Pseudo Consistency (ALGO)Backbone=Devstral-Small-25052026.060.8030.704
Consistency (VR)Model=Qwen3-32B2026.060.7990.843
Top-5 + Tests + PseudoModel=Qwen3-Coder-30B2026.060.7960.852
Top-5 + Tests + PseudoModel=Qwen3-14B2026.060.7920.87
Top-5 + Tests + PseudoBackbone=Qwen3-14B2026.060.7920.87
Max EntropyModel=Qwen3-Coder-30B2026.060.780.806
Consistency (VR)Model=Qwen3-14B2026.060.7760.798
Consistency (VR)Backbone=Qwen3-14B2026.060.7760.798
Symb. ClusteringModel=Qwen3-14B2026.060.7640.813
Symb. ClusteringBackbone=Qwen3-14B2026.060.7640.813
Consistency (VR)Model=DeepSeek-Coder-V22026.060.7590.797
Consistency (VR)Backbone=DeepSeek-Coder-V22026.060.7590.797
Top-5 + Tests + PseudoModel=Qwen3-32B2026.060.7590.884
Top-5 + Tests + PseudoModel=Devstral-Small-25052026.060.7520.739
Top-5 + Tests + PseudoBackbone=Devstral-Small-25052026.060.7520.739
Top-5 + Tests + PseudoModel=DeepSeek-Coder-V22026.060.7490.814
Top-5 + Tests + PseudoBackbone=DeepSeek-Coder-V22026.060.7490.814
Generated Tests (FUNC)Model=Qwen3-Coder-30B2026.060.7490.809
Generated Tests (FUNC)Model=Devstral-Small-25052026.060.7470.748
Generated Tests (FUNC)Backbone=Devstral-Small-25052026.060.7470.748
Generated Tests (FUNC)Model=Qwen3-14B2026.060.7450.787
Consistency (VR)Model=Devstral-Small-25052026.060.7450.737
Generated Tests (FUNC)Backbone=Qwen3-14B2026.060.7450.787
Consistency (VR)Backbone=Devstral-Small-25052026.060.7450.737
Generated Tests (FUNC)Model=Qwen3-32B2026.060.7430.804
Consistency (BLEU)Model=Qwen3-14B2026.060.7420.776
Consistency (BLEU)Backbone=Qwen3-14B2026.060.7420.776
Consistency (BLEU)Model=Qwen3-32B2026.060.7390.797
Consistency (BLEU)Model=Qwen3-Coder-30B2026.060.7360.781
Max EntropyModel=Qwen3-14B2026.060.7240.767
Consistency (BLEU)Model=DeepSeek-Coder-V22026.060.7210.767
Consistency (BLEU)Backbone=DeepSeek-Coder-V22026.060.7210.767
Pseudo Consistency (ALGO)Model=Qwen3-Coder-30B2026.060.7210.838
Max ProbabilityModel=Qwen3-14B2026.060.720.718
Mean ProbabilityModel=Qwen3-14B2026.060.7160.751
Pseudo Consistency (ALGO)Model=Qwen3-14B2026.060.7130.833
Pseudo Consistency (ALGO)Backbone=Qwen3-14B2026.060.7130.833
Generated Tests (FUNC)Model=DeepSeek-Coder-V22026.060.7080.763
Generated Tests (FUNC)Backbone=DeepSeek-Coder-V22026.060.7080.763
Top-5 Entropy (LEX)Model=Devstral-Small-25052026.060.7010.651
Top-5 Entropy (LEX)Backbone=Devstral-Small-25052026.060.7010.651
Mean EntropyModel=Qwen3-14B2026.060.6940.7
Mean EntropyBackbone=Qwen3-14B2026.060.6940.7
Pseudo Consistency (ALGO)Model=DeepSeek-Coder-V22026.060.6910.802
Pseudo Consistency (ALGO)Backbone=DeepSeek-Coder-V22026.060.6910.802
Consistency (VR)Model=Qwen3-Coder-30B2026.060.6890.733
Pseudo Consistency (ALGO)Model=Qwen3-32B2026.060.6710.753
Mean EntropyModel=Qwen3-Coder-30B2026.060.670.71
Symb. ClusteringModel=Qwen3-32B2026.060.6650.776
Top-5 Entropy (LEX)Model=DeepSeek-Coder-V22026.060.6370.668
Top-5 Entropy (LEX)Backbone=DeepSeek-Coder-V22026.060.6370.668
Max ProbabilityModel=Qwen3-Coder-30B2026.060.6330.692
Consistency (BLEU)Model=Devstral-Small-25052026.060.6140.624
Consistency (BLEU)Backbone=Devstral-Small-25052026.060.6140.624
Mean ProbabilityModel=Qwen3-Coder-30B2026.060.6120.659
Symb. ClusteringModel=Qwen3-Coder-30B2026.060.6120.697
Top-5 Entropy (LEX)Model=Qwen3-32B2026.060.5730.644
Symb. ClusteringModel=DeepSeek-Coder-V22026.060.5660.643
Symb. ClusteringBackbone=DeepSeek-Coder-V22026.060.5660.643
Mean ProbabilityModel=Devstral-Small-25052026.060.5150.512
Max ProbabilityModel=DeepSeek-Coder-V22026.060.510.53
Max EntropyModel=Devstral-Small-25052026.060.5080.469
Max ProbabilityModel=Devstral-Small-25052026.060.5040.51
Max EntropyModel=DeepSeek-Coder-V22026.060.5030.534
Mean ProbabilityModel=DeepSeek-Coder-V22026.060.4990.528
Mean EntropyModel=Devstral-Small-25052026.060.4910.461
Mean EntropyBackbone=Devstral-Small-25052026.060.4910.461
Symb. ClusteringModel=Devstral-Small-25052026.060.4840.506
Symb. ClusteringBackbone=Devstral-Small-25052026.060.4840.506
Mean EntropyModel=DeepSeek-Coder-V22026.060.4830.532
Mean EntropyBackbone=DeepSeek-Coder-V22026.060.4830.532
Mean ProbabilityModel=Qwen3-32B2026.060.4690.602
Max ProbabilityModel=Qwen3-32B2026.060.4670.605
Max EntropyModel=Qwen3-32B2026.060.460.559
Mean EntropyModel=Qwen3-32B2026.060.450.561