Long-context language generation on RepoBench-P
4.46Average Acceptance LengthLongSpec
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| LongSpecBackbone=LC-13B, Temperature=02025.02 | 4.46 | 96.96 | 3.26 | |
| LongSpecBackbone=LC-7B, Temperature=02025.02 | 4.03 | 115.27 | 2.7 | |
| LongSpecBackbone=V-7B, Temperature=02025.02 | 3.86 | 110.76 | 2.38 | |
| LongSpecBackbone=V-13B, Temperature=02025.02 | 3.59 | 77.22 | 2.65 | |
| LongSpecBackbone=L-8B, Temperature=02025.02 | 3.39 | 91.28 | 1.69 | |
| MagicDecBackbone=V-13B, Temperature=02025.02 | 2.94 | 36.66 | 1.26 | |
| MagicDecBackbone=LC-13B, Temperature=02025.02 | 2.85 | 35.67 | 1.2 | |
| PLDBackbone=LC-7B, Temperature=02025.02 | 2.71 | 89.22 | 2.09 | |
| MagicDecBackbone=L-8B, Temperature=02025.02 | 2.61 | 44.39 | 0.82 | |
| MagicDecBackbone=V-7B, Temperature=02025.02 | 2.57 | 48.75 | 1.05 | |
| MagicDecBackbone=LC-7B, Temperature=02025.02 | 2.57 | 44.13 | 1.03 | |
| PLDBackbone=V-7B, Temperature=02025.02 | 2.23 | 74.15 | 1.59 | |
| PLDBackbone=LC-13B, Temperature=02025.02 | 1.8 | 41.07 | 1.38 | |
| PLDBackbone=L-8B, Temperature=02025.02 | 1.38 | 45.7 | 0.85 | |
| PLDBackbone=V-13B, Temperature=02025.02 | 1.32 | 30.6 | 1.05 | |
| Vanilla HFBackbone=V-7B, Temperature=02025.02 | 1 | 19.18 | — | |
| Vanilla FABackbone=V-7B, Temperature=02025.02 | 1 | 46.61 | 1 | |
| Vanilla HFBackbone=V-13B, Temperature=02025.02 | 1 | 13.44 | — | |
| Vanilla FABackbone=V-13B, Temperature=02025.02 | 1 | 29.14 | 1 | |
| Vanilla HFBackbone=LC-7B, Temperature=02025.02 | 1 | 17.02 | — | |
| Vanilla FABackbone=LC-7B, Temperature=02025.02 | 1 | 42.69 | 1 | |
| Vanilla HFBackbone=LC-13B, Temperature=02025.02 | 1 | 13.85 | — | |
| Vanilla FABackbone=LC-13B, Temperature=02025.02 | 1 | 29.74 | 1 | |
| Vanilla HFBackbone=L-8B, Temperature=02025.02 | 1 | 22.77 | — | |
| Vanilla FABackbone=L-8B, Temperature=02025.02 | 1 | 54.08 | 1 |