Fill-in-the-Middle Code Completion on Delulu
62.4EM (%)ORPO
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ORPOModel Size=7B, Truncation Strategy=untruncated2026.06 | 62.4 | 85.6 | |
| ORPOModel Size=7B, Truncation Strategy=+trunc2026.06 | 62.4 | 85.5 | |
| SFTModel Size=7B, Truncation Strategy=+trunc2026.06 | 61.8 | 85 | |
| SFTModel Size=7B, Truncation Strategy=untruncated2026.06 | 61.7 | 85.2 | |
| ORPOModel Size=3B, Truncation Strategy=+trunc2026.06 | 59.9 | 84.1 | |
| ORPOModel Size=3B, Truncation Strategy=untruncated2026.06 | 59.7 | 84.1 | |
| SFTModel Size=3B, Truncation Strategy=+trunc2026.06 | 59.2 | 83.5 | |
| SFTModel Size=3B, Truncation Strategy=untruncated2026.06 | 58.7 | 83.3 | |
| baseModel Size=3B, Truncation Strategy=+trunc2026.06 | 52 | 79.8 | |
| baseModel Size=7B, Truncation Strategy=+trunc2026.06 | 48.8 | 77 | |
| baseModel Size=3B, Truncation Strategy=untruncated2026.06 | 45.9 | 69.5 | |
| baseModel Size=7B, Truncation Strategy=untruncated2026.06 | 42.9 | 62.8 | |
| DPOModel Size=7B, Truncation Strategy=+trunc2026.06 | 25.8 | 62.4 | |
| DPOModel Size=7B, Truncation Strategy=untruncated2026.06 | 24.2 | 36.2 | |
| DPOModel Size=3B, Truncation Strategy=+trunc2026.06 | 16.9 | 55.4 | |
| DPOModel Size=3B, Truncation Strategy=untruncated2026.06 | 13.3 | 28.8 |