Code Generation on LiveCodeBench (pass@1, pass@3, pass@5)
1,784Pass@1Bifrost
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| BifrostModel=Llama-3.1-8B-Instruct2026.02 | 1,784 | 2,098 | 2,636 | |
| PaprikaModel=Llama-3.1-8B-Instruct2026.02 | 1,498 | 2,030 | 2,261 | |
| BifrostModel=Llama-3.2-3B-Instruct2026.02 | 1,350 | 1,893 | 2,148 | |
| ICLModel=Llama-3.1-8B-Instruct2026.02 | 1,280 | 1,474 | 1,822 | |
| BoTModel=Llama-3.1-8B-Instruct2026.02 | 1,041 | 1,639 | 1,886 | |
| ICLModel=Llama-3.2-3B-Instruct2026.02 | 1,023 | 1,400 | 1,545 | |
| BaseModel=Llama-3.1-8B-Instruct2026.02 | 986 | 1,236 | 1,575 | |
| BaseModel=Llama-3.2-3B-Instruct2026.02 | 868 | 1,220 | 1,398 | |
| PaprikaModel=Llama-3.2-3B-Instruct2026.02 | 859 | 1,233 | 1,420 | |
| RISEModel=Llama-3.1-8B-Instruct2026.02 | 723 | 1,426 | 1,795 | |
| ReflexionModel=Llama-3.1-8B-Instruct2026.02 | 636 | 807 | 852 | |
| BoTModel=Llama-3.2-3B-Instruct2026.02 | 570 | 893 | 989 | |
| RISEModel=Llama-3.2-3B-Instruct2026.02 | 439 | 902 | 1,205 | |
| BifrostModel=Llama-3.2-1B-Instruct2026.02 | 405 | 774 | 955 | |
| DoTModel=Llama-3.2-3B-Instruct2026.02 | 273 | 295 | 295 | |
| ICLModel=Llama-3.2-1B-Instruct2026.02 | 184 | 363 | 466 | |
| DoTModel=Llama-3.1-8B-Instruct2026.02 | 159 | 182 | 205 | |
| PaprikaModel=Llama-3.2-1B-Instruct2026.02 | 130 | 255 | 330 | |
| ReflexionModel=Llama-3.2-3B-Instruct2026.02 | 125 | 159 | 170 | |
| BaseModel=Llama-3.2-1B-Instruct2026.02 | 107 | 211 | 284 | |
| RISEModel=Llama-3.2-1B-Instruct2026.02 | 66 | 172 | 250 | |
| Gemini-2.5-ProInstitution=Google2026.03 | 62.01 | — | — | |
| ReflexiCoder-8B (Multiple)Institution=Ours, setup=full iterative reasoning-reflection setup2026.03 | 54.12 | — | — | |
| ReflexiCoder-8B (Single)Institution=Ours, setup=single-attempt without system prompt2026.03 | 52.21 | — | — | |
| Claude-Sonnet-4.5Institution=Anthropic2026.03 | 50.78 | — | — | |
| GPT-5.1Institution=OpenAI2026.03 | 48.03 | — | — | |
| GPT-4.1Institution=OpenAI2026.03 | 42.77 | — | — | |
| Qwen3-8BInstitution=Alibaba2026.03 | 37.75 | — | — | |
| DeepCoder-14B-PreviewInstitution=rLLM2026.03 | 34.05 | — | — | |
| BoTModel=Llama-3.2-1B-Instruct2026.02 | 27 | 69 | 102 | |
| BF16Model=DeepSeek-R1-Distill-Qwen-7B, Bit-width=BF16, Maximum number of generation tokens=32,7682025.12 | 23.08 | — | — | |
| DeepCoder-1.5B-PreviewInstitution=rLLM2026.03 | 22.94 | — | — | |
| Seed-Coder-8B-InstructInstitution=ByteDance2026.03 | 21.03 | — | — | |
| KIVIModel=DeepSeek-R1-Distill-Qwen-7B, Bit-width=KV4, Maximum number of generation tokens=32,7682025.12 | 20.88 | — | — | |
| MixKVQModel=DeepSeek-R1-Distill-Qwen-7B, Bit-width=C3.4, Maximum number of generation tokens=32,7682025.12 | 20.33 | — | — | |
| KVQuantModel=DeepSeek-R1-Distill-Qwen-7B, Bit-width=KV4, Maximum number of generation tokens=32,7682025.12 | 19.44 | — | — | |
| KVTunerModel=DeepSeek-R1-Distill-Qwen-7B, Bit-width=C3.92, Maximum number of generation tokens=32,7682025.12 | 19.23 | — | — | |
| RotateKVModel=DeepSeek-R1-Distill-Qwen-7B, Bit-width=KV4, Maximum number of generation tokens=32,7682025.12 | 18.33 | — | — | |
| KIVIModel=DeepSeek-R1-Distill-Qwen-7B, Bit-width=KV2, Maximum number of generation tokens=32,7682025.12 | 16.48 | — | — | |
| Qwen2.5-Coder-7B-InstructInstitution=Alibaba2026.03 | 13.86 | — | — | |
| BifrostModel=Llama-3.2-3B-Instruct2026.02 | 12.73 | 16.99 | 18.75 | |
| ReflexionModel=Llama-3.2-1B-Instruct2026.02 | 11 | 11 | 11 | |
| PaprikaModel=Llama-3.2-3B-Instruct2026.02 | 9.33 | 12.37 | 13.7 | |
| ICLModel=Llama-3.2-3B-Instruct2026.02 | 8.75 | 11.93 | 13.64 | |
| DeepSeek-Coder-7B-InstructInstitution=DeepSeek2026.03 | 8.24 | — | — | |
| CodeGemma-7B-ITInstitution=Google2026.03 | 8.12 | — | — | |
| KVQuantModel=DeepSeek-R1-Distill-Qwen-7B, Bit-width=KV2, Maximum number of generation tokens=32,7682025.12 | 5.49 | — | — | |
| RISEModel=Llama-3.2-3B-Instruct2026.02 | 4.81 | 8.85 | 10.74 | |
| BoTModel=Llama-3.2-3B-Instruct2026.02 | 4.67 | 7.41 | 8.52 | |
| CodeLlama-7b-InstructInstitution=Meta2026.03 | 1.19 | — | — | |
| DoTModel=Llama-3.2-1B-Instruct2026.02 | 0 | 0 | 0 |