Code Generation on ReplicationBench
27.8Pass@3MTL (I)
Evaluation results
| Method | Links | |
|---|---|---|
| MTL (I)Model=DeepSeek V3.2, Method=MTL (I)2026.04 | 27.8 | |
| Zero-shotModel=DeepSeek V3.2, Method=Zero-shot2026.04 | 26.7 | |
| Zero-shotModel=Qwen3-Coder-480B-A35B-Instruct, Method=Zero-shot2026.04 | 21.1 | |
| MTL (I)Model=Qwen3-Coder-480B-A35B-Instruct, Method=MTL (I)2026.04 | 21.1 | |
| AgentKB#Memories=5,8992026.04 | 20 | |
| MTL (I)Model=GPT-5-mini, Method=MTL (I)2026.04 | 18.9 | |
| MTL#Memories=4312026.04 | 18.9 | |
| MTL (S)Model=GPT-5-mini, Method=MTL (S)2026.04 | 13.3 | |
| ReasoningBank#Memories=972026.04 | 13.3 | |
| MTL (T)Model=GPT-5-mini, Method=MTL (T)2026.04 | 12.2 | |
| Zero-shotModel=GPT-5-mini, Method=Zero-shot2026.04 | 11.1 | |
| MTL (W)Model=GPT-5-mini, Method=MTL (W)2026.04 | 11.1 | |
| Zeroshot#Memories=-2026.04 | 11.1 |