Code Generation on HumanEval, HumanEval+, MBPP, MBPP+, Bigcode (test)
48.2HumanEvalSelectIT
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| SelectITBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 48.2 | 38.4 | 60.6 | 51.9 | 23 | 44.4 | |
| InstructDiffBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 48.2 | 40.9 | 60.6 | 50 | 25.7 | 45.1 | |
| Resp LenMaxBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 47 | 37.8 | 57.4 | 48.4 | 24.2 | 43 | |
| Inst/RespMinBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 47 | 38.4 | 56.1 | 46.3 | 22.8 | 42.1 | |
| EntropyMinBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 46.3 | 39 | 58.7 | 46.3 | 23.4 | 42.7 | |
| ALLBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 44.5 | 35.4 | 59.8 | 49.7 | 25.8 | 43 | |
| PPLMidBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 44.5 | 36.6 | 56.9 | 46.8 | 21.6 | 41.3 | |
| Inst/RespMaxBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 43.9 | 38.4 | 59 | 49.2 | 21.9 | 42.5 | |
| SuperfilteringBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 43.3 | 34.8 | 54.2 | 44.2 | 20.6 | 39.4 | |
| Inst LenMaxBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 42.1 | 35.4 | 57.7 | 46.3 | 23.6 | 41 | |
| RandomBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 42.1 | 33.5 | 57.7 | 47.9 | 22.9 | 40.8 | |
| PPLMinBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 41.5 | 34.1 | 56.1 | 46.3 | 23.4 | 40.3 | |
| EntropyMidBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 41.5 | 34.1 | 59.3 | 48.1 | 23.8 | 41.4 | |
| EntropyMaxBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 40.9 | 34.8 | 57.9 | 48.4 | 21.1 | 40.6 | |
| IFDBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 40.9 | 34.1 | 51.6 | 40.7 | 18.2 | 37.1 | |
| Base ModelBackbone=LLaMA3-8B, Selection Ratio=0%2026.01 | 40.2 | 33.5 | 55.3 | 46.3 | 22.5 | 39.6 | |
| PPLMaxBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 40.2 | 34.1 | 55.6 | 47.4 | 21 | 39.7 | |
| ZIPBackbone=LLaMA3-8B, Selection Ratio=20%2026.01 | 38.4 | 32.9 | 59.3 | 50.3 | 16 | 39.4 |