List your company
ResearchFormerly WizWand

Long-context language generation on RepoBench-P

4.46Average Acceptance LengthLongSpec
0.86161.79582.733.6642Feb 24, 2025
Updated 6mo ago

Evaluation results

MethodLinks
LongSpecBackbone=LC-13B, Temperature=02025.024.4696.963.26
LongSpecBackbone=LC-7B, Temperature=02025.024.03115.272.7
LongSpecBackbone=V-7B, Temperature=02025.023.86110.762.38
LongSpecBackbone=V-13B, Temperature=02025.023.5977.222.65
LongSpecBackbone=L-8B, Temperature=02025.023.3991.281.69
MagicDecBackbone=V-13B, Temperature=02025.022.9436.661.26
MagicDecBackbone=LC-13B, Temperature=02025.022.8535.671.2
PLDBackbone=LC-7B, Temperature=02025.022.7189.222.09
MagicDecBackbone=L-8B, Temperature=02025.022.6144.390.82
MagicDecBackbone=V-7B, Temperature=02025.022.5748.751.05
MagicDecBackbone=LC-7B, Temperature=02025.022.5744.131.03
PLDBackbone=V-7B, Temperature=02025.022.2374.151.59
PLDBackbone=LC-13B, Temperature=02025.021.841.071.38
PLDBackbone=L-8B, Temperature=02025.021.3845.70.85
PLDBackbone=V-13B, Temperature=02025.021.3230.61.05
Vanilla HFBackbone=V-7B, Temperature=02025.02119.18—
Vanilla FABackbone=V-7B, Temperature=02025.02146.611
Vanilla HFBackbone=V-13B, Temperature=02025.02113.44—
Vanilla FABackbone=V-13B, Temperature=02025.02129.141
Vanilla HFBackbone=LC-7B, Temperature=02025.02117.02—
Vanilla FABackbone=LC-7B, Temperature=02025.02142.691
Vanilla HFBackbone=LC-13B, Temperature=02025.02113.85—
Vanilla FABackbone=LC-13B, Temperature=02025.02129.741
Vanilla HFBackbone=L-8B, Temperature=02025.02122.77—
Vanilla FABackbone=L-8B, Temperature=02025.02154.081