Code Generation on MBPP 3-shot (pass@1 accuracy)
41.6Pass@1 AccuracySAS
Evaluation results
| Method | Links | |
|---|---|---|
| SASGeneration Length (L)=128, Block Size (B)=162026.06 | 41.6 | |
| SASGeneration Length (L)=256, Block Size (B)=2562026.06 | 41 | |
| SASGeneration Length (L)=256, Block Size (B)=642026.06 | 40.7 | |
| SASGeneration Length (L)=512, Block Size (B)=5122026.06 | 40.7 | |
| SASGeneration Length (L)=128, Block Size (B)=322026.06 | 40.5 | |
| Confidence (Top-1)Generation Length (L)=128, Block Size (B)=322026.06 | 40 | |
| SASGeneration Length (L)=256, Block Size (B)=322026.06 | 40 | |
| Confidence (Top-1)Generation Length (L)=256, Block Size (B)=642026.06 | 39.6 | |
| EntropyGeneration Length (L)=128, Block Size (B)=162026.06 | 39.6 | |
| Confidence (Top-1)Generation Length (L)=256, Block Size (B)=2562026.06 | 39.5 | |
| SASGeneration Length (L)=512, Block Size (B)=1282026.06 | 39.5 | |
| SASGeneration Length (L)=512, Block Size (B)=642026.06 | 39.5 | |
| EntropyGeneration Length (L)=512, Block Size (B)=642026.06 | 39.3 | |
| Confidence (Top-1)Generation Length (L)=512, Block Size (B)=1282026.06 | 39.2 | |
| MarginGeneration Length (L)=128, Block Size (B)=322026.06 | 39 | |
| EntropyGeneration Length (L)=512, Block Size (B)=5122026.06 | 39 | |
| EntropyGeneration Length (L)=512, Block Size (B)=1282026.06 | 39 | |
| Confidence (Top-1)Generation Length (L)=128, Block Size (B)=162026.06 | 38.8 | |
| Confidence (Top-1)Generation Length (L)=512, Block Size (B)=642026.06 | 38.8 | |
| MarginGeneration Length (L)=128, Block Size (B)=162026.06 | 38.8 | |
| MarginGeneration Length (L)=512, Block Size (B)=1282026.06 | 38.8 | |
| EntropyGeneration Length (L)=128, Block Size (B)=322026.06 | 38.8 | |
| EntropyGeneration Length (L)=256, Block Size (B)=2562026.06 | 38.8 | |
| Confidence (Top-1)Generation Length (L)=256, Block Size (B)=322026.06 | 38.7 | |
| Confidence (Top-1)Generation Length (L)=512, Block Size (B)=5122026.06 | 38.5 | |
| MarginGeneration Length (L)=512, Block Size (B)=5122026.06 | 38.5 | |
| EntropyGeneration Length (L)=256, Block Size (B)=642026.06 | 38.5 | |
| MarginGeneration Length (L)=256, Block Size (B)=642026.06 | 38 | |
| MarginGeneration Length (L)=512, Block Size (B)=642026.06 | 38 | |
| MarginGeneration Length (L)=256, Block Size (B)=322026.06 | 37.8 | |
| EntropyGeneration Length (L)=256, Block Size (B)=322026.06 | 37.8 | |
| SASGeneration Length (L)=128, Block Size (B)=1282026.06 | 37.5 | |
| Confidence (Top-1)Generation Length (L)=128, Block Size (B)=1282026.06 | 37.3 | |
| MarginGeneration Length (L)=256, Block Size (B)=2562026.06 | 36.3 | |
| MarginGeneration Length (L)=128, Block Size (B)=1282026.06 | 33.3 | |
| EntropyGeneration Length (L)=128, Block Size (B)=1282026.06 | 32.3 |