Code Output Prediction on CRUXEval-O
79.1Pass@1MiMo-V2 Flash
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MiMo-V2 FlashShots=1-shot, Chain-of-Thought (CoT)=true, # Total Params=309B, # Active Params=15.1B2026.05 | 79.1 | — | |
| Qwen3.5Shots=1-shot, Chain-of-Thought (CoT)=true, # Total Params=35B, # Active Params=2.6B2026.05 | 71.9 | — | |
| LAGUNA XS.2Shots=1-shot, Chain-of-Thought (CoT)=true, # Total Params=33.4B, # Active Params=3B2026.05 | 71.7 | — | |
| Nemotron-3-NanoShots=1-shot, Chain-of-Thought (CoT)=true, # Total Params=31.6B, # Active Params=3.6B2026.05 | 63.4 | — | |
| Gemma-4Shots=1-shot, Chain-of-Thought (CoT)=true, # Total Params=25.2B, # Active Params=3.8B2026.05 | 60 | — | |
| LoRAModel=DeepseekCoder-7B, #Params=11.8M2024.05 | 49.8 | — | |
| DeepSeekCoderParameters=33B2024.02 | 48.6 | 61.6 | |
| LoPAModel=DeepseekCoder-7B, #Params=6.35M2024.05 | 48 | — | |
| StarCoder2Parameters=15B2024.02 | 47.1 | 59.5 | |
| PTModel=DeepseekCoder-7B, #Params=41.0K2024.05 | 44.8 | — | |
| None (Zero-shot)Model=DeepseekCoder-7B, #Params=02024.05 | 44 | — | |
| DeepSeekCoderParameters=6.7B2024.02 | 43.5 | 54.8 | |
| LoRAModel=Phi2-2.7B, #Params=7.9M2024.05 | 42.5 | — | |
| LoPAModel=Phi3-3.8B, #Params=5.3M2024.05 | 42.5 | — | |
| CodeLlamaParameters=34B2024.02 | 42.4 | 55.9 | |
| S-IDPGModel=DeepseekCoder-7B, #Params=32.1M2024.05 | 41.5 | — | |
| LoRAModel=Phi3-3.8B, #Params=6.3M2024.05 | 41.2 | — | |
| LoRAModel=Llama3-8B, #Params=9.4M2024.05 | 40.5 | — | |
| LoPAModel=Llama3-8B, #Params=6.4M2024.05 | 39.8 | — | |
| CodeLlamaParameters=13B2024.02 | 39.7 | 53.9 | |
| FFTModel=Phi3-3.8B, #Params=3.8B2024.05 | 39.5 | — | |
| None (Zero-shot)Model=Phi3-3.8B, #Params=02024.05 | 39.5 | — | |
| S-IDPGModel=Phi3-3.8B, #Params=24.2M2024.05 | 39.5 | — | |
| LoPAModel=Phi2-2.7B, #Params=4.76M2024.05 | 37.2 | — | |
| FFTModel=Phi2-2.7B, #Params=2.7B2024.05 | 37 | — | |
| StarCoder2Parameters=7B2024.02 | 36 | 52 | |
| LoRAModel=DeepseekCoder-1.3B, #Params=4.7M2024.05 | 36 | — | |
| S-IDPGModel=Llama3-8B, #Params=32.1M2024.05 | 35.2 | — | |
| FFTModel=DeepseekCoder-1.3B, #Params=1.3B2024.05 | 34.8 | — | |
| LoPAModel=DeepseekCoder-1.3B, #Params=4.2M2024.05 | 34.5 | — | |
| StarCoder2Parameters=3B2024.02 | 34.2 | 48.4 | |
| CodeLlamaParameters=7B2024.02 | 34.2 | 48.4 | |
| StarCoderBaseParameters=15B2024.02 | 34.2 | 47.1 | |
| PTModel=Phi2-2.7B, #Params=25.6K2024.05 | 34 | — | |
| None (Zero-shot)Model=Phi2-2.7B, #Params=02024.05 | 33 | — | |
| S-IDPGModel=Phi2-2.7B, #Params=20.3M2024.05 | 33 | — | |
| StarCoderBaseParameters=7B2024.02 | 32.2 | 44.9 | |
| PTModel=Llama3-8B, #Params=41.0K2024.05 | 32 | — | |
| PTModel=Phi3-3.8B, #Params=30.7K2024.05 | 31.5 | — | |
| None (Zero-shot)Model=Llama3-8B, #Params=02024.05 | 31.5 | — | |
| PTModel=DeepseekCoder-1.3B, #Params=20.5K2024.05 | 31.2 | — | |
| DeepSeekCoderParameters=1.3B2024.02 | 31 | 43.4 | |
| None (Zero-shot)Model=DeepseekCoder-1.3B, #Params=02024.05 | 29.8 | — | |
| S-IDPGModel=DeepseekCoder-1.3B, #Params=16.3M2024.05 | 28.5 | — | |
| StarCoderBaseParameters=3B2024.02 | 27.4 | 40.9 | |
| StableCodeParameters=3B2024.02 | 26.7 | 43.5 | |
| FFTModel=CodeGen-350M, #Params=350M2024.05 | 19.5 | — | |
| LoPAModel=CodeGen-350M, #Params=4.4M2024.05 | 18.5 | — | |
| LoRAModel=CodeGen-350M, #Params=1.3M2024.05 | 18.2 | — | |
| PTModel=CodeGen-350M, #Params=10.2K2024.05 | 15.8 | — | |
| None (Zero-shot)Model=CodeGen-350M, #Params=02024.05 | 15 | — | |
| S-IDPGModel=CodeGen-350M, #Params=8.5M2024.05 | 13.2 | — |