Code Generation on LiveCodeBench October 2024 to January 2025 (test)
46.1Pass@1ThinkCoder
Evaluation Results
| Method | Links | |
|---|---|---|
| ThinkCoderBackbone=GPT-4-Turbo, Self-refinement budget (n)=22024.12 | 46.1 | |
| ThinkCoderBackbone=GPT-4-Turbo, Self-refinement budget (n)=52024.12 | 44.9 | |
| ThinkCoderBackbone=GPT-4-Turbo, Self-refinement budget (n)=12024.12 | 43.7 | |
| ThinkCoderBackbone=GPT-4o, Self-refinement budget (n)=52024.12 | 41.3 | |
| ThinkCoderBackbone=GPT-4o, Self-refinement budget (n)=22024.12 | 38.3 | |
| ThinkCoderBackbone=GPT-4o, Self-refinement budget (n)=12024.12 | 35.3 | |
| ThinkCoderBackbone=GPT-4-Turbo, Self-refinement budget (n)=02024.12 | 33.4 | |
| ThinkCoderBackbone=GPT-4o, Self-refinement budget (n)=02024.12 | 22.5 |