Uncertainty Estimation on APPS Intro
0.813AUROCTop-5 Entropy (LEX)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Top-5 Entropy (LEX)Model=Qwen3-14B2026.06 | 0.813 | 0.828 | |
| Top-5 Entropy (LEX)Backbone=Qwen3-14B2026.06 | 0.813 | 0.828 | |
| Top-5 Entropy (LEX)Model=Qwen3-Coder-30B2026.06 | 0.807 | 0.822 | |
| Pseudo Consistency (ALGO)Model=Devstral-Small-25052026.06 | 0.803 | 0.704 | |
| Pseudo Consistency (ALGO)Backbone=Devstral-Small-25052026.06 | 0.803 | 0.704 | |
| Consistency (VR)Model=Qwen3-32B2026.06 | 0.799 | 0.843 | |
| Top-5 + Tests + PseudoModel=Qwen3-Coder-30B2026.06 | 0.796 | 0.852 | |
| Top-5 + Tests + PseudoModel=Qwen3-14B2026.06 | 0.792 | 0.87 | |
| Top-5 + Tests + PseudoBackbone=Qwen3-14B2026.06 | 0.792 | 0.87 | |
| Max EntropyModel=Qwen3-Coder-30B2026.06 | 0.78 | 0.806 | |
| Consistency (VR)Model=Qwen3-14B2026.06 | 0.776 | 0.798 | |
| Consistency (VR)Backbone=Qwen3-14B2026.06 | 0.776 | 0.798 | |
| Symb. ClusteringModel=Qwen3-14B2026.06 | 0.764 | 0.813 | |
| Symb. ClusteringBackbone=Qwen3-14B2026.06 | 0.764 | 0.813 | |
| Consistency (VR)Model=DeepSeek-Coder-V22026.06 | 0.759 | 0.797 | |
| Consistency (VR)Backbone=DeepSeek-Coder-V22026.06 | 0.759 | 0.797 | |
| Top-5 + Tests + PseudoModel=Qwen3-32B2026.06 | 0.759 | 0.884 | |
| Top-5 + Tests + PseudoModel=Devstral-Small-25052026.06 | 0.752 | 0.739 | |
| Top-5 + Tests + PseudoBackbone=Devstral-Small-25052026.06 | 0.752 | 0.739 | |
| Top-5 + Tests + PseudoModel=DeepSeek-Coder-V22026.06 | 0.749 | 0.814 | |
| Top-5 + Tests + PseudoBackbone=DeepSeek-Coder-V22026.06 | 0.749 | 0.814 | |
| Generated Tests (FUNC)Model=Qwen3-Coder-30B2026.06 | 0.749 | 0.809 | |
| Generated Tests (FUNC)Model=Devstral-Small-25052026.06 | 0.747 | 0.748 | |
| Generated Tests (FUNC)Backbone=Devstral-Small-25052026.06 | 0.747 | 0.748 | |
| Generated Tests (FUNC)Model=Qwen3-14B2026.06 | 0.745 | 0.787 | |
| Consistency (VR)Model=Devstral-Small-25052026.06 | 0.745 | 0.737 | |
| Generated Tests (FUNC)Backbone=Qwen3-14B2026.06 | 0.745 | 0.787 | |
| Consistency (VR)Backbone=Devstral-Small-25052026.06 | 0.745 | 0.737 | |
| Generated Tests (FUNC)Model=Qwen3-32B2026.06 | 0.743 | 0.804 | |
| Consistency (BLEU)Model=Qwen3-14B2026.06 | 0.742 | 0.776 | |
| Consistency (BLEU)Backbone=Qwen3-14B2026.06 | 0.742 | 0.776 | |
| Consistency (BLEU)Model=Qwen3-32B2026.06 | 0.739 | 0.797 | |
| Consistency (BLEU)Model=Qwen3-Coder-30B2026.06 | 0.736 | 0.781 | |
| Max EntropyModel=Qwen3-14B2026.06 | 0.724 | 0.767 | |
| Consistency (BLEU)Model=DeepSeek-Coder-V22026.06 | 0.721 | 0.767 | |
| Consistency (BLEU)Backbone=DeepSeek-Coder-V22026.06 | 0.721 | 0.767 | |
| Pseudo Consistency (ALGO)Model=Qwen3-Coder-30B2026.06 | 0.721 | 0.838 | |
| Max ProbabilityModel=Qwen3-14B2026.06 | 0.72 | 0.718 | |
| Mean ProbabilityModel=Qwen3-14B2026.06 | 0.716 | 0.751 | |
| Pseudo Consistency (ALGO)Model=Qwen3-14B2026.06 | 0.713 | 0.833 | |
| Pseudo Consistency (ALGO)Backbone=Qwen3-14B2026.06 | 0.713 | 0.833 | |
| Generated Tests (FUNC)Model=DeepSeek-Coder-V22026.06 | 0.708 | 0.763 | |
| Generated Tests (FUNC)Backbone=DeepSeek-Coder-V22026.06 | 0.708 | 0.763 | |
| Top-5 Entropy (LEX)Model=Devstral-Small-25052026.06 | 0.701 | 0.651 | |
| Top-5 Entropy (LEX)Backbone=Devstral-Small-25052026.06 | 0.701 | 0.651 | |
| Mean EntropyModel=Qwen3-14B2026.06 | 0.694 | 0.7 | |
| Mean EntropyBackbone=Qwen3-14B2026.06 | 0.694 | 0.7 | |
| Pseudo Consistency (ALGO)Model=DeepSeek-Coder-V22026.06 | 0.691 | 0.802 | |
| Pseudo Consistency (ALGO)Backbone=DeepSeek-Coder-V22026.06 | 0.691 | 0.802 | |
| Consistency (VR)Model=Qwen3-Coder-30B2026.06 | 0.689 | 0.733 | |
| Pseudo Consistency (ALGO)Model=Qwen3-32B2026.06 | 0.671 | 0.753 | |
| Mean EntropyModel=Qwen3-Coder-30B2026.06 | 0.67 | 0.71 | |
| Symb. ClusteringModel=Qwen3-32B2026.06 | 0.665 | 0.776 | |
| Top-5 Entropy (LEX)Model=DeepSeek-Coder-V22026.06 | 0.637 | 0.668 | |
| Top-5 Entropy (LEX)Backbone=DeepSeek-Coder-V22026.06 | 0.637 | 0.668 | |
| Max ProbabilityModel=Qwen3-Coder-30B2026.06 | 0.633 | 0.692 | |
| Consistency (BLEU)Model=Devstral-Small-25052026.06 | 0.614 | 0.624 | |
| Consistency (BLEU)Backbone=Devstral-Small-25052026.06 | 0.614 | 0.624 | |
| Mean ProbabilityModel=Qwen3-Coder-30B2026.06 | 0.612 | 0.659 | |
| Symb. ClusteringModel=Qwen3-Coder-30B2026.06 | 0.612 | 0.697 | |
| Top-5 Entropy (LEX)Model=Qwen3-32B2026.06 | 0.573 | 0.644 | |
| Symb. ClusteringModel=DeepSeek-Coder-V22026.06 | 0.566 | 0.643 | |
| Symb. ClusteringBackbone=DeepSeek-Coder-V22026.06 | 0.566 | 0.643 | |
| Mean ProbabilityModel=Devstral-Small-25052026.06 | 0.515 | 0.512 | |
| Max ProbabilityModel=DeepSeek-Coder-V22026.06 | 0.51 | 0.53 | |
| Max EntropyModel=Devstral-Small-25052026.06 | 0.508 | 0.469 | |
| Max ProbabilityModel=Devstral-Small-25052026.06 | 0.504 | 0.51 | |
| Max EntropyModel=DeepSeek-Coder-V22026.06 | 0.503 | 0.534 | |
| Mean ProbabilityModel=DeepSeek-Coder-V22026.06 | 0.499 | 0.528 | |
| Mean EntropyModel=Devstral-Small-25052026.06 | 0.491 | 0.461 | |
| Mean EntropyBackbone=Devstral-Small-25052026.06 | 0.491 | 0.461 | |
| Symb. ClusteringModel=Devstral-Small-25052026.06 | 0.484 | 0.506 | |
| Symb. ClusteringBackbone=Devstral-Small-25052026.06 | 0.484 | 0.506 | |
| Mean EntropyModel=DeepSeek-Coder-V22026.06 | 0.483 | 0.532 | |
| Mean EntropyBackbone=DeepSeek-Coder-V22026.06 | 0.483 | 0.532 | |
| Mean ProbabilityModel=Qwen3-32B2026.06 | 0.469 | 0.602 | |
| Max ProbabilityModel=Qwen3-32B2026.06 | 0.467 | 0.605 | |
| Max EntropyModel=Qwen3-32B2026.06 | 0.46 | 0.559 | |
| Mean EntropyModel=Qwen3-32B2026.06 | 0.45 | 0.561 |