Research Replication on ReplicationBench
17.8Pass@1MTL (I)
Evaluation results
| Method | Links | |
|---|---|---|
| MTL (I)Model Backbone=DeepSeek V3.22026.04 | 17.8 | |
| ZeroShotModel Backbone=DeepSeek V3.22026.04 | 14.4 | |
| MTL (I)Model Backbone=Qwen3-Coder-480B-A35B-Instruct2026.04 | 13 | |
| ZeroShotModel Backbone=Qwen3-Coder-480B-A35B-Instruct2026.04 | 12.6 | |
| MTL (I)Model Backbone=GPT-5-mini2026.04 | 11.9 | |
| MTL (W)Model Backbone=GPT-5-mini2026.04 | 8.1 | |
| MTL (S)Model Backbone=GPT-5-mini2026.04 | 7.8 | |
| MTL (T)Model Backbone=GPT-5-mini2026.04 | 6.3 | |
| ZeroShotModel Backbone=GPT-5-mini2026.04 | 5.9 |