OpenAPI completion on OpenAPI completion benchmark
45Correctness (Max)Code Llama 7B
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Code Llama 7BFine-tuning context size=4096 tokens2024.05 | 45 | 84 | 32 | 63.1 | |
| Code Llama 7BFine-tuning context size=5120 tokens2024.05 | 42 | 79 | 26.8 | 57.1 | |
| Code Llama 7BDocument splitting=true2024.05 | 42 | 76 | 34 | 69.1 | |
| Code LlamaModel size=7B2024.05 | 36 | 64 | 31.1 | 60.7 | |
| Code LlamaModel size=13B, Evaluation context window=1024-4096 tokens2024.05 | 34 | 68 | 30.2 | 64 | |
| GitHub Copilot2024.05 | 29 | 68 | 29 | 68 |