Uncertainty Estimation on APPS Interview
81AUROCTop-5 + Tests + Pseudo
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Top-5 + Tests + PseudoModel=Qwen3-14B2026.06 | 81 | 72.5 | |
| Top-5 + Tests + PseudoBackbone=Qwen3-14B2026.06 | 81 | 72.5 | |
| Top-5 Entropy (LEX)Model=Qwen3-14B2026.06 | 79.8 | 61.1 | |
| Top-5 Entropy (LEX)Backbone=Qwen3-14B2026.06 | 79.8 | 61.1 | |
| Generated Tests (FUNC)Model=Qwen3-14B2026.06 | 75.5 | 54.9 | |
| Generated Tests (FUNC)Backbone=Qwen3-14B2026.06 | 75.5 | 54.9 | |
| Consistency (VR)Model=Qwen3-14B2026.06 | 74.8 | 52.5 | |
| Consistency (VR)Backbone=Qwen3-14B2026.06 | 74.8 | 52.5 | |
| Consistency (VR)Model=DeepSeek-Coder-V22026.06 | 74.7 | 52.2 | |
| Top-5 + Tests + PseudoModel=DeepSeek-Coder-V22026.06 | 74.7 | 55.8 | |
| Consistency (VR)Backbone=DeepSeek-Coder-V22026.06 | 74.7 | 52.2 | |
| Top-5 + Tests + PseudoBackbone=DeepSeek-Coder-V22026.06 | 74.7 | 55.8 | |
| Top-5 + Tests + PseudoModel=Devstral-Small-25052026.06 | 73.4 | 44.9 | |
| Top-5 + Tests + PseudoBackbone=Devstral-Small-25052026.06 | 73.4 | 44.9 | |
| Consistency (BLEU)Model=Qwen3-14B2026.06 | 72.3 | 49.5 | |
| Consistency (BLEU)Backbone=Qwen3-14B2026.06 | 72.3 | 49.5 | |
| Generated Tests (FUNC)Model=Devstral-Small-25052026.06 | 71.2 | 39.2 | |
| Generated Tests (FUNC)Backbone=Devstral-Small-25052026.06 | 71.2 | 39.2 | |
| Consistency (BLEU)Model=DeepSeek-Coder-V22026.06 | 70.3 | 47 | |
| Consistency (BLEU)Backbone=DeepSeek-Coder-V22026.06 | 70.3 | 47 | |
| Generated Tests (FUNC)Model=DeepSeek-Coder-V22026.06 | 69.2 | 50.1 | |
| Generated Tests (FUNC)Backbone=DeepSeek-Coder-V22026.06 | 69.2 | 50.1 | |
| Mean EntropyModel=Qwen3-14B2026.06 | 69.1 | 41.4 | |
| Mean EntropyBackbone=Qwen3-14B2026.06 | 69.1 | 41.4 | |
| Symb. ClusteringModel=Qwen3-14B2026.06 | 69 | 52.2 | |
| Symb. ClusteringBackbone=Qwen3-14B2026.06 | 69 | 52.2 | |
| Consistency (VR)Model=Devstral-Small-25052026.06 | 67.5 | 28.1 | |
| Consistency (VR)Backbone=Devstral-Small-25052026.06 | 67.5 | 28.1 | |
| Max EntropyModel=Qwen3-14B2026.06 | 67.2 | 41.1 | |
| Pseudo Consistency (ALGO)Model=DeepSeek-Coder-V22026.06 | 65.3 | 50 | |
| Pseudo Consistency (ALGO)Backbone=DeepSeek-Coder-V22026.06 | 65.3 | 50 | |
| Max EntropyModel=DeepSeek-Coder-V22026.06 | 65.2 | 34.9 | |
| Pseudo Consistency (ALGO)Model=Devstral-Small-25052026.06 | 64.6 | 43.3 | |
| Pseudo Consistency (ALGO)Backbone=Devstral-Small-25052026.06 | 64.6 | 43.3 | |
| Top-5 Entropy (LEX)Model=DeepSeek-Coder-V22026.06 | 63.4 | 36.8 | |
| Top-5 Entropy (LEX)Backbone=DeepSeek-Coder-V22026.06 | 63.4 | 36.8 | |
| Top-5 Entropy (LEX)Model=Devstral-Small-25052026.06 | 62.7 | 23.9 | |
| Top-5 Entropy (LEX)Backbone=Devstral-Small-25052026.06 | 62.7 | 23.9 | |
| Max ProbabilityModel=Qwen3-14B2026.06 | 62.5 | 51.3 | |
| Consistency (BLEU)Model=Devstral-Small-25052026.06 | 60.9 | 24.6 | |
| Consistency (BLEU)Backbone=Devstral-Small-25052026.06 | 60.9 | 24.6 | |
| Pseudo Consistency (ALGO)Model=Qwen3-14B2026.06 | 60.4 | 58.8 | |
| Pseudo Consistency (ALGO)Backbone=Qwen3-14B2026.06 | 60.4 | 58.8 | |
| Mean ProbabilityModel=Qwen3-14B2026.06 | 58.2 | 40.8 | |
| Max ProbabilityModel=Devstral-Small-25052026.06 | 56.7 | 23.7 | |
| Mean EntropyModel=Devstral-Small-25052026.06 | 55.8 | 18.9 | |
| Mean EntropyBackbone=Devstral-Small-25052026.06 | 55.8 | 18.9 | |
| Mean ProbabilityModel=Devstral-Small-25052026.06 | 54.9 | 20.8 | |
| Max ProbabilityModel=DeepSeek-Coder-V22026.06 | 54.6 | 34.1 | |
| Max EntropyModel=Devstral-Small-25052026.06 | 54.4 | 20 | |
| Mean EntropyModel=DeepSeek-Coder-V22026.06 | 52.8 | 27.7 | |
| Mean EntropyBackbone=DeepSeek-Coder-V22026.06 | 52.8 | 27.7 | |
| Mean ProbabilityModel=DeepSeek-Coder-V22026.06 | 51.9 | 31.8 | |
| Symb. ClusteringModel=DeepSeek-Coder-V22026.06 | 47.3 | 27.7 | |
| Symb. ClusteringBackbone=DeepSeek-Coder-V22026.06 | 47.3 | 27.7 | |
| Symb. ClusteringModel=Devstral-Small-25052026.06 | 46.1 | 19 | |
| Symb. ClusteringBackbone=Devstral-Small-25052026.06 | 46.1 | 19 | |
| Top-5 + Tests + PseudoModel=Qwen3-Coder-30B2026.06 | 0.8 | 0.614 | |
| Generated Tests (FUNC)Model=Qwen3-32B2026.06 | 0.77 | 0.571 | |
| Top-5 + Tests + PseudoModel=Qwen3-32B2026.06 | 0.768 | 0.603 | |
| Consistency (VR)Model=Qwen3-32B2026.06 | 0.762 | 0.587 | |
| Top-5 Entropy (LEX)Model=Qwen3-Coder-30B2026.06 | 0.756 | 0.58 | |
| Generated Tests (FUNC)Model=Qwen3-Coder-30B2026.06 | 0.738 | 0.576 | |
| Consistency (BLEU)Model=Qwen3-Coder-30B2026.06 | 0.725 | 0.544 | |
| Max EntropyModel=Qwen3-Coder-30B2026.06 | 0.722 | 0.546 | |
| Consistency (BLEU)Model=Qwen3-32B2026.06 | 0.705 | 0.479 | |
| Consistency (VR)Model=Qwen3-Coder-30B2026.06 | 0.667 | 0.483 | |
| Mean EntropyModel=Qwen3-Coder-30B2026.06 | 0.655 | 0.458 | |
| Max ProbabilityModel=Qwen3-Coder-30B2026.06 | 0.643 | 0.522 | |
| Pseudo Consistency (ALGO)Model=Qwen3-32B2026.06 | 0.642 | 0.448 | |
| Mean ProbabilityModel=Qwen3-Coder-30B2026.06 | 0.619 | 0.518 | |
| Top-5 Entropy (LEX)Model=Qwen3-32B2026.06 | 0.616 | 0.369 | |
| Symb. ClusteringModel=Qwen3-32B2026.06 | 0.61 | 0.477 | |
| Pseudo Consistency (ALGO)Model=Qwen3-Coder-30B2026.06 | 0.61 | 0.672 | |
| Symb. ClusteringModel=Qwen3-Coder-30B2026.06 | 0.55 | 0.434 | |
| Mean EntropyModel=Qwen3-32B2026.06 | 0.532 | 0.291 | |
| Max ProbabilityModel=Qwen3-32B2026.06 | 0.528 | 0.323 | |
| Mean ProbabilityModel=Qwen3-32B2026.06 | 0.52 | 0.316 | |
| Max EntropyModel=Qwen3-32B2026.06 | 0.491 | 0.283 |