RepoBench
Benchmarks
TaskDatasetSOTA resultTrendResults
RepoBench-P LongBench
52.05Pass@1
30
RepoBench-P
4.46Average Acceptance Length
25
RepoBench-P
0.731Similarity
17
RepoBench
3.57Speedup
12
RepoBench >8k
51.24Edit Sim
12
RepoBench 4k-8k
53.3Edit Similarity
12
RepoBench 0-4k
52.82Edit Similarity
12
RepoBench-P
1.83MAT
11
RepoBench (test)
65.9Exact-match Accuracy
7
RepoBench-P Python, XF-Random
64.5Execution Match (EM)
6
RepoBench-P Python XF-First
52.4Exact Match (EM)
6
RepoBench
25.3Pass@1
6
RepoBench P
15.04Score
5
RepoBench
48.92Pass@k Score
1