Uncertainty Estimation on MBPP
0.791AUROCTop-5 + Tests + Pseudo
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Top-5 + Tests + PseudoModel=Devstral-Small-25052026.06 | 0.791 | 0.785 | |
| Top-5 + Tests + PseudoBackbone=Devstral-Small-25052026.06 | 0.791 | 0.785 | |
| Generated Tests (FUNC)Model=Devstral-Small-25052026.06 | 0.776 | 0.735 | |
| Generated Tests (FUNC)Backbone=Devstral-Small-25052026.06 | 0.776 | 0.735 | |
| Top-5 + Tests + PseudoModel=Qwen3-Coder-30B2026.06 | 0.748 | 0.767 | |
| Top-5 + Tests + PseudoModel=Qwen3-14B2026.06 | 0.746 | 0.761 | |
| Top-5 + Tests + PseudoBackbone=Qwen3-14B2026.06 | 0.746 | 0.761 | |
| Generated Tests (FUNC)Model=DeepSeek-Coder-V22026.06 | 0.741 | 0.736 | |
| Generated Tests (FUNC)Backbone=DeepSeek-Coder-V22026.06 | 0.741 | 0.736 | |
| Top-5 + Tests + PseudoModel=DeepSeek-Coder-V22026.06 | 0.74 | 0.755 | |
| Top-5 + Tests + PseudoBackbone=DeepSeek-Coder-V22026.06 | 0.74 | 0.755 | |
| Generated Tests (FUNC)Model=Qwen3-Coder-30B2026.06 | 0.731 | 0.722 | |
| Generated Tests (FUNC)Model=Qwen3-14B2026.06 | 0.73 | 0.702 | |
| Generated Tests (FUNC)Backbone=Qwen3-14B2026.06 | 0.73 | 0.702 | |
| Top-5 + Tests + PseudoModel=Qwen3-32B2026.06 | 0.727 | 0.738 | |
| Pseudo Consistency (ALGO)Model=DeepSeek-Coder-V22026.06 | 0.722 | 0.636 | |
| Pseudo Consistency (ALGO)Backbone=DeepSeek-Coder-V22026.06 | 0.722 | 0.636 | |
| Generated Tests (FUNC)Model=Qwen3-32B2026.06 | 0.711 | 0.724 | |
| Consistency (VR)Model=Qwen3-32B2026.06 | 0.663 | 0.713 | |
| Symb. ClusteringModel=Qwen3-14B2026.06 | 0.661 | 0.691 | |
| Symb. ClusteringBackbone=Qwen3-14B2026.06 | 0.661 | 0.691 | |
| Consistency (VR)Model=Qwen3-14B2026.06 | 0.648 | 0.684 | |
| Consistency (VR)Backbone=Qwen3-14B2026.06 | 0.648 | 0.684 | |
| Pseudo Consistency (ALGO)Model=Devstral-Small-25052026.06 | 0.647 | 0.62 | |
| Pseudo Consistency (ALGO)Backbone=Devstral-Small-25052026.06 | 0.647 | 0.62 | |
| Consistency (BLEU)Model=Qwen3-14B2026.06 | 0.64 | 0.673 | |
| Consistency (BLEU)Backbone=Qwen3-14B2026.06 | 0.64 | 0.673 | |
| Max ProbabilityModel=DeepSeek-Coder-V22026.06 | 0.617 | 0.634 | |
| Consistency (VR)Model=DeepSeek-Coder-V22026.06 | 0.616 | 0.639 | |
| Top-5 Entropy (LEX)Model=DeepSeek-Coder-V22026.06 | 0.616 | 0.653 | |
| Consistency (VR)Backbone=DeepSeek-Coder-V22026.06 | 0.616 | 0.639 | |
| Top-5 Entropy (LEX)Backbone=DeepSeek-Coder-V22026.06 | 0.616 | 0.653 | |
| Pseudo Consistency (ALGO)Model=Qwen3-14B2026.06 | 0.615 | 0.65 | |
| Pseudo Consistency (ALGO)Backbone=Qwen3-14B2026.06 | 0.615 | 0.65 | |
| Top-5 Entropy (LEX)Model=Devstral-Small-25052026.06 | 0.614 | 0.653 | |
| Top-5 Entropy (LEX)Backbone=Devstral-Small-25052026.06 | 0.614 | 0.653 | |
| Consistency (VR)Model=Devstral-Small-25052026.06 | 0.611 | 0.642 | |
| Consistency (VR)Backbone=Devstral-Small-25052026.06 | 0.611 | 0.642 | |
| Pseudo Consistency (ALGO)Model=Qwen3-32B2026.06 | 0.611 | 0.656 | |
| Consistency (BLEU)Model=Qwen3-32B2026.06 | 0.603 | 0.664 | |
| Pseudo Consistency (ALGO)Model=Qwen3-Coder-30B2026.06 | 0.602 | 0.66 | |
| Max EntropyModel=DeepSeek-Coder-V22026.06 | 0.601 | 0.62 | |
| Consistency (BLEU)Model=Qwen3-Coder-30B2026.06 | 0.6 | 0.619 | |
| Top-5 Entropy (LEX)Model=Qwen3-Coder-30B2026.06 | 0.599 | 0.629 | |
| Consistency (BLEU)Model=Devstral-Small-25052026.06 | 0.597 | 0.621 | |
| Consistency (BLEU)Backbone=Devstral-Small-25052026.06 | 0.597 | 0.621 | |
| Mean ProbabilityModel=DeepSeek-Coder-V22026.06 | 0.596 | 0.612 | |
| Consistency (BLEU)Model=DeepSeek-Coder-V22026.06 | 0.596 | 0.641 | |
| Consistency (BLEU)Backbone=DeepSeek-Coder-V22026.06 | 0.596 | 0.641 | |
| Top-5 Entropy (LEX)Model=Qwen3-32B2026.06 | 0.592 | 0.657 | |
| Symb. ClusteringModel=Qwen3-Coder-30B2026.06 | 0.592 | 0.616 | |
| Mean EntropyModel=Qwen3-Coder-30B2026.06 | 0.585 | 0.627 | |
| Mean ProbabilityModel=Qwen3-32B2026.06 | 0.58 | 0.658 | |
| Max ProbabilityModel=Qwen3-32B2026.06 | 0.573 | 0.66 | |
| Top-5 Entropy (LEX)Model=Qwen3-14B2026.06 | 0.57 | 0.616 | |
| Top-5 Entropy (LEX)Backbone=Qwen3-14B2026.06 | 0.57 | 0.616 | |
| Max EntropyModel=Qwen3-Coder-30B2026.06 | 0.57 | 0.607 | |
| Symb. ClusteringModel=DeepSeek-Coder-V22026.06 | 0.565 | 0.639 | |
| Symb. ClusteringBackbone=DeepSeek-Coder-V22026.06 | 0.565 | 0.639 | |
| Symb. ClusteringModel=Qwen3-32B2026.06 | 0.56 | 0.66 | |
| Mean EntropyModel=DeepSeek-Coder-V22026.06 | 0.544 | 0.611 | |
| Mean EntropyBackbone=DeepSeek-Coder-V22026.06 | 0.544 | 0.611 | |
| Mean EntropyModel=Qwen3-32B2026.06 | 0.544 | 0.617 | |
| Max ProbabilityModel=Qwen3-Coder-30B2026.06 | 0.536 | 0.609 | |
| Max ProbabilityModel=Devstral-Small-25052026.06 | 0.535 | 0.57 | |
| Max EntropyModel=Qwen3-14B2026.06 | 0.532 | 0.569 | |
| Max EntropyModel=Devstral-Small-25052026.06 | 0.532 | 0.563 | |
| Max EntropyModel=Qwen3-32B2026.06 | 0.532 | 0.632 | |
| Mean ProbabilityModel=Devstral-Small-25052026.06 | 0.529 | 0.561 | |
| Mean EntropyModel=Devstral-Small-25052026.06 | 0.528 | 0.558 | |
| Mean EntropyBackbone=Devstral-Small-25052026.06 | 0.528 | 0.558 | |
| Consistency (VR)Model=Qwen3-Coder-30B2026.06 | 0.528 | 0.56 | |
| Mean EntropyModel=Qwen3-14B2026.06 | 0.526 | 0.565 | |
| Mean EntropyBackbone=Qwen3-14B2026.06 | 0.526 | 0.565 | |
| Max ProbabilityModel=Qwen3-14B2026.06 | 0.521 | 0.564 | |
| Mean ProbabilityModel=Qwen3-14B2026.06 | 0.518 | 0.555 | |
| Symb. ClusteringModel=Devstral-Small-25052026.06 | 0.514 | 0.575 | |
| Symb. ClusteringBackbone=Devstral-Small-25052026.06 | 0.514 | 0.575 | |
| Mean ProbabilityModel=Qwen3-Coder-30B2026.06 | 0.51 | 0.61 |