Code Generation on HumanEval (Calibration and Uncertainty Metrics)
81AUROCVEROIC
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| VEROICsetting=CMIS, inference_path=LLaMA-3.1-8B -> LLaMA-3.1-70B2026.04 | 81 | 48 | 0.16 | 0.473 | 5.7 | |
| RISK PREDICTORsetting=CMIS, inference_path=LLaMA-3.1-8B -> LLaMA-3.1-70B2026.04 | 80 | 46 | 0.166 | 0.482 | 6.5 | |
| CONFIDENCE-THRESHOLDsetting=CMIS, inference_path=LLaMA-3.1-8B -> LLaMA-3.1-70B2026.04 | 78 | 42 | 0.175 | 0.505 | 7.4 | |
| OUTPUT-TRIGGERsetting=CMIS, inference_path=LLaMA-3.1-8B -> LLaMA-3.1-70B2026.04 | 77 | 41 | 0.171 | 0.495 | 7.2 | |
| PERIODICsetting=CMIS, inference_path=LLaMA-3.1-8B -> LLaMA-3.1-70B2026.04 | 74 | 38 | 0.182 | 0.522 | 7.6 | |
| RANDOMsetting=CMIS, inference_path=LLaMA-3.1-8B -> LLaMA-3.1-70B2026.04 | 72 | 33 | 0.197 | 0.529 | 8.4 | |
| FIXEDsetting=CMIS, inference_path=LLaMA-3.1-8B -> LLaMA-3.1-70B2026.04 | 66 | 26 | 0.205 | 0.584 | 9.6 |