Code Input Prediction on CRUXEval-I
67.5Pass@1MiMo-V2 Flash
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MiMo-V2 FlashShots=1-shot, Chain-of-Thought (CoT)=true, # Total Params=309B, # Active Params=15.1B2026.05 | 67.5 | — | |
| Qwen3.5Shots=1-shot, Chain-of-Thought (CoT)=true, # Total Params=35B, # Active Params=2.6B2026.05 | 66 | — | |
| Nemotron-3-NanoShots=1-shot, Chain-of-Thought (CoT)=true, # Total Params=31.6B, # Active Params=3.6B2026.05 | 63.2 | — | |
| LAGUNA XS.2Shots=1-shot, Chain-of-Thought (CoT)=true, # Total Params=33.4B, # Active Params=3B2026.05 | 61.9 | — | |
| Gemma-4Shots=1-shot, Chain-of-Thought (CoT)=true, # Total Params=25.2B, # Active Params=3.8B2026.05 | 54.7 | — | |
| LoPAModel=DeepseekCoder-7B, #Params=6.35M2024.05 | 50 | — | |
| StarCoder2Parameters=15B2024.02 | 48.1 | 66.9 | |
| LoRAModel=DeepseekCoder-7B, #Params=11.8M2024.05 | 47.5 | — | |
| CodeLlamaParameters=34B2024.02 | 47.2 | 66.6 | |
| DeepSeekCoderParameters=33B2024.02 | 46.5 | 64.9 | |
| PTModel=DeepseekCoder-7B, #Params=41.0K2024.05 | 45.8 | — | |
| LoRAModel=Llama3-8B, #Params=9.4M2024.05 | 45.5 | — | |
| FFTModel=DeepseekCoder-1.3B, #Params=1.3B2024.05 | 45 | — | |
| LoPAModel=DeepseekCoder-1.3B, #Params=4.2M2024.05 | 43 | — | |
| LoPAModel=Phi2-2.7B, #Params=4.76M2024.05 | 43 | — | |
| CodeLlamaParameters=13B2024.02 | 42.5 | 62 | |
| LoPAModel=Phi3-3.8B, #Params=5.3M2024.05 | 42.2 | — | |
| DeepSeekCoderParameters=6.7B2024.02 | 41.9 | 62.7 | |
| LoRAModel=Phi2-2.7B, #Params=7.9M2024.05 | 41.5 | — | |
| PTModel=DeepseekCoder-1.3B, #Params=20.5K2024.05 | 41.2 | — | |
| LoPAModel=Llama3-8B, #Params=6.4M2024.05 | 41.2 | — | |
| S-IDPGModel=DeepseekCoder-7B, #Params=32.1M2024.05 | 40.5 | — | |
| FFTModel=Phi2-2.7B, #Params=2.7B2024.05 | 40.2 | — | |
| None (Zero-shot)Model=DeepseekCoder-7B, #Params=02024.05 | 39.3 | — | |
| FFTModel=Phi3-3.8B, #Params=3.8B2024.05 | 39.2 | — | |
| LoRAModel=Phi3-3.8B, #Params=6.3M2024.05 | 38 | — | |
| PTModel=Llama3-8B, #Params=41.0K2024.05 | 37.5 | — | |
| CodeLlamaParameters=7B2024.02 | 35.9 | 52.9 | |
| LoRAModel=DeepseekCoder-1.3B, #Params=4.7M2024.05 | 35.5 | — | |
| PTModel=Phi2-2.7B, #Params=25.6K2024.05 | 35 | — | |
| S-IDPGModel=Phi2-2.7B, #Params=20.3M2024.05 | 35 | — | |
| StarCoder2Parameters=7B2024.02 | 34.6 | 53.5 | |
| LoPAModel=CodeGen-350M, #Params=4.4M2024.05 | 34.5 | — | |
| None (Zero-shot)Model=Phi3-3.8B, #Params=02024.05 | 33.8 | — | |
| StableCodeParameters=3B2024.02 | 33.5 | 53.3 | |
| None (Zero-shot)Model=Phi2-2.7B, #Params=02024.05 | 33.5 | — | |
| PTModel=Phi3-3.8B, #Params=30.7K2024.05 | 33.5 | — | |
| FFTModel=CodeGen-350M, #Params=350M2024.05 | 33 | — | |
| PTModel=CodeGen-350M, #Params=10.2K2024.05 | 32.8 | — | |
| StarCoder2Parameters=3B2024.02 | 32.7 | 50.1 | |
| StarCoderBaseParameters=15B2024.02 | 31.3 | 49.2 | |
| LoRAModel=CodeGen-350M, #Params=1.3M2024.05 | 31 | — | |
| S-IDPGModel=Phi3-3.8B, #Params=24.2M2024.05 | 31 | — | |
| StarCoderBaseParameters=7B2024.02 | 29.7 | 47.3 | |
| S-IDPGModel=Llama3-8B, #Params=32.1M2024.05 | 29.2 | — | |
| DeepSeekCoderParameters=1.3B2024.02 | 27.8 | 44.7 | |
| StarCoderBaseParameters=3B2024.02 | 27.1 | 43.7 | |
| None (Zero-shot)Model=Llama3-8B, #Params=02024.05 | 27 | — | |
| None (Zero-shot)Model=DeepseekCoder-1.3B, #Params=02024.05 | 26.8 | — | |
| S-IDPGModel=DeepseekCoder-1.3B, #Params=16.3M2024.05 | 26 | — | |
| None (Zero-shot)Model=CodeGen-350M, #Params=02024.05 | 20.8 | — | |
| S-IDPGModel=CodeGen-350M, #Params=8.5M2024.05 | 16.9 | — |