Uncertainty Estimation on HumanEval
0.886AUROCTop-5 + Tests + Pseudo
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Top-5 + Tests + PseudoModel=Devstral-Small-25052026.06 | 0.886 | 0.99 | |
| Top-5 + Tests + PseudoBackbone=Devstral-Small-25052026.06 | 0.886 | 0.99 | |
| Generated Tests (FUNC)Model=DeepSeek-Coder-V22026.06 | 0.861 | 0.952 | |
| Generated Tests (FUNC)Backbone=DeepSeek-Coder-V22026.06 | 0.861 | 0.952 | |
| Top-5 + Tests + PseudoModel=Qwen3-14B2026.06 | 0.852 | 0.983 | |
| Top-5 + Tests + PseudoBackbone=Qwen3-14B2026.06 | 0.852 | 0.983 | |
| Top-5 + Tests + PseudoModel=DeepSeek-Coder-V22026.06 | 0.842 | 0.957 | |
| Top-5 + Tests + PseudoBackbone=DeepSeek-Coder-V22026.06 | 0.842 | 0.957 | |
| Generated Tests (FUNC)Model=Qwen3-14B2026.06 | 0.822 | 0.958 | |
| Generated Tests (FUNC)Backbone=Qwen3-14B2026.06 | 0.822 | 0.958 | |
| Generated Tests (FUNC)Model=Devstral-Small-25052026.06 | 0.819 | 0.94 | |
| Generated Tests (FUNC)Backbone=Devstral-Small-25052026.06 | 0.819 | 0.94 | |
| Pseudo Consistency (ALGO)Model=Devstral-Small-25052026.06 | 0.768 | 0.939 | |
| Pseudo Consistency (ALGO)Backbone=Devstral-Small-25052026.06 | 0.768 | 0.939 | |
| Pseudo Consistency (ALGO)Model=DeepSeek-Coder-V22026.06 | 0.759 | 0.832 | |
| Pseudo Consistency (ALGO)Backbone=DeepSeek-Coder-V22026.06 | 0.759 | 0.832 | |
| Generated Tests (FUNC)Model=Qwen3-32B2026.06 | 0.756 | 0.938 | |
| Top-5 Entropy (LEX)Model=Devstral-Small-25052026.06 | 0.749 | 0.934 | |
| Top-5 Entropy (LEX)Backbone=Devstral-Small-25052026.06 | 0.749 | 0.934 | |
| Top-5 + Tests + PseudoModel=Qwen3-Coder-30B2026.06 | 0.746 | 0.98 | |
| Consistency (VR)Model=Qwen3-14B2026.06 | 0.741 | 0.955 | |
| Consistency (VR)Backbone=Qwen3-14B2026.06 | 0.741 | 0.955 | |
| Top-5 + Tests + PseudoModel=Qwen3-32B2026.06 | 0.734 | 0.944 | |
| Generated Tests (FUNC)Model=Qwen3-Coder-30B2026.06 | 0.734 | 0.973 | |
| Top-5 Entropy (LEX)Model=Qwen3-14B2026.06 | 0.729 | 0.952 | |
| Top-5 Entropy (LEX)Backbone=Qwen3-14B2026.06 | 0.729 | 0.952 | |
| Pseudo Consistency (ALGO)Model=Qwen3-14B2026.06 | 0.718 | 0.947 | |
| Pseudo Consistency (ALGO)Backbone=Qwen3-14B2026.06 | 0.718 | 0.947 | |
| Mean EntropyModel=Qwen3-14B2026.06 | 0.715 | 0.941 | |
| Mean EntropyBackbone=Qwen3-14B2026.06 | 0.715 | 0.941 | |
| Consistency (BLEU)Model=Qwen3-Coder-30B2026.06 | 0.705 | 0.981 | |
| Consistency (BLEU)Model=Qwen3-14B2026.06 | 0.694 | 0.945 | |
| Consistency (BLEU)Backbone=Qwen3-14B2026.06 | 0.694 | 0.945 | |
| Max ProbabilityModel=Devstral-Small-25052026.06 | 0.68 | 0.902 | |
| Max EntropyModel=Qwen3-Coder-30B2026.06 | 0.678 | 0.977 | |
| Mean ProbabilityModel=Devstral-Small-25052026.06 | 0.675 | 0.889 | |
| Symb. ClusteringModel=Qwen3-14B2026.06 | 0.673 | 0.943 | |
| Symb. ClusteringBackbone=Qwen3-14B2026.06 | 0.673 | 0.943 | |
| Pseudo Consistency (ALGO)Model=Qwen3-Coder-30B2026.06 | 0.671 | 0.973 | |
| Symb. ClusteringModel=Devstral-Small-25052026.06 | 0.668 | 0.893 | |
| Symb. ClusteringBackbone=Devstral-Small-25052026.06 | 0.668 | 0.893 | |
| Max ProbabilityModel=Qwen3-Coder-30B2026.06 | 0.667 | 0.96 | |
| Mean ProbabilityModel=Qwen3-Coder-30B2026.06 | 0.662 | 0.951 | |
| Symb. ClusteringModel=Qwen3-32B2026.06 | 0.654 | 0.922 | |
| Symb. ClusteringModel=DeepSeek-Coder-V22026.06 | 0.646 | 0.895 | |
| Symb. ClusteringBackbone=DeepSeek-Coder-V22026.06 | 0.646 | 0.895 | |
| Consistency (VR)Model=Devstral-Small-25052026.06 | 0.644 | 0.875 | |
| Consistency (VR)Backbone=Devstral-Small-25052026.06 | 0.644 | 0.875 | |
| Consistency (VR)Model=DeepSeek-Coder-V22026.06 | 0.635 | 0.877 | |
| Consistency (VR)Backbone=DeepSeek-Coder-V22026.06 | 0.635 | 0.877 | |
| Max ProbabilityModel=Qwen3-14B2026.06 | 0.63 | 0.935 | |
| Pseudo Consistency (ALGO)Model=Qwen3-32B2026.06 | 0.621 | 0.915 | |
| Mean ProbabilityModel=Qwen3-14B2026.06 | 0.619 | 0.902 | |
| Max EntropyModel=Devstral-Small-25052026.06 | 0.619 | 0.881 | |
| Consistency (BLEU)Model=Devstral-Small-25052026.06 | 0.616 | 0.885 | |
| Consistency (BLEU)Backbone=Devstral-Small-25052026.06 | 0.616 | 0.885 | |
| Top-5 Entropy (LEX)Model=Qwen3-32B2026.06 | 0.613 | 0.902 | |
| Mean EntropyModel=Devstral-Small-25052026.06 | 0.612 | 0.876 | |
| Mean EntropyBackbone=Devstral-Small-25052026.06 | 0.612 | 0.876 | |
| Max EntropyModel=Qwen3-14B2026.06 | 0.601 | 0.898 | |
| Top-5 Entropy (LEX)Model=Qwen3-Coder-30B2026.06 | 0.591 | 0.967 | |
| Consistency (BLEU)Model=DeepSeek-Coder-V22026.06 | 0.586 | 0.85 | |
| Consistency (BLEU)Backbone=DeepSeek-Coder-V22026.06 | 0.586 | 0.85 | |
| Mean EntropyModel=Qwen3-Coder-30B2026.06 | 0.581 | 0.957 | |
| Consistency (VR)Model=Qwen3-Coder-30B2026.06 | 0.578 | 0.962 | |
| Consistency (VR)Model=Qwen3-32B2026.06 | 0.574 | 0.898 | |
| Max ProbabilityModel=Qwen3-32B2026.06 | 0.549 | 0.899 | |
| Mean ProbabilityModel=Qwen3-32B2026.06 | 0.53 | 0.9 | |
| Max EntropyModel=Qwen3-32B2026.06 | 0.523 | 0.889 | |
| Symb. ClusteringModel=Qwen3-Coder-30B2026.06 | 0.517 | 0.96 | |
| Consistency (BLEU)Model=Qwen3-32B2026.06 | 0.516 | 0.885 | |
| Mean EntropyModel=Qwen3-32B2026.06 | 0.49 | 0.868 | |
| Top-5 Entropy (LEX)Model=DeepSeek-Coder-V22026.06 | 0.466 | 0.827 | |
| Top-5 Entropy (LEX)Backbone=DeepSeek-Coder-V22026.06 | 0.466 | 0.827 | |
| Max ProbabilityModel=DeepSeek-Coder-V22026.06 | 0.425 | 0.788 | |
| Max EntropyModel=DeepSeek-Coder-V22026.06 | 0.412 | 0.826 | |
| Mean ProbabilityModel=DeepSeek-Coder-V22026.06 | 0.393 | 0.767 | |
| Mean EntropyModel=DeepSeek-Coder-V22026.06 | 0.376 | 0.798 | |
| Mean EntropyBackbone=DeepSeek-Coder-V22026.06 | 0.376 | 0.798 |