Code Generation on MBPP (Score %)
66.17MBPP ScorePiSSA
Evaluation Results
| Method | Links | |
|---|---|---|
| PiSSAModel=Gemma-7B2024.04 | 66.17 | |
| Qwen2-7BParameters=7B2024.07 | 65.9 | |
| LoRAModel=Gemma-7B, Initialization=Gaussian2024.04 | 65.58 | |
| LoRAModel=Gemma-7B, Initialization=Kaiming2024.04 | 65.25 | |
| PiSSAModel=Mistral-7B2024.04 | 62.55 | |
| LoRAModel=Mistral-7B, Initialization=Gaussian2024.04 | 58.46 | |
| LoRAModel=Mistral-7B, Initialization=Kaiming2024.04 | 58.39 | |
| Full FTModel=Gemma-7B2024.04 | 55.67 | |
| Llama-3-8BParameters=8B2024.07 | 53.9 | |
| Qwen2-1.5Bnumber of parameters=1.5B2024.07 | 51.9 | |
| Qwen1.5-7BParameters=7B2024.07 | 51.6 | |
| Full FTModel=Mistral-7B2024.04 | 51.46 | |
| Mistral-7BParameters=7B2024.07 | 51.1 | |
| Gemma-7BParameters=7B2024.07 | 50.6 | |
| PRINMIXalpha=1/162025.06 | 50.2 | |
| PRINMIXalpha=1/642025.06 | 50 | |
| Alignedalpha=12025.06 | 48.9 | |
| LoRAalpha=1/162025.06 | 47.7 | |
| PRINMIX-LoRAalpha=1/642025.06 | 47.7 | |
| Backbonealpha=12025.06 | 46 | |
| KEELTraining=SFT, Evaluation Setting=0-shot2026.01 | 42.6 | |
| Qwen2-0.5Bnumber of parameters=0.5B2024.07 | 37.8 | |
| Pre-LNTraining=SFT, Evaluation Setting=0-shot2026.01 | 37.6 | |
| Qwen-1.5 7BRole=Teacher2024.07 | 37.4 | |
| PiSSAModel=LLAMA 2-7B2024.04 | 37.24 | |
| Full FTModel=LLAMA 2-7B2024.04 | 35.59 | |
| LoRAModel=LLAMA 2-7B, Initialization=Gaussian2024.04 | 35.5 | |
| LoRAModel=LLAMA 2-7B, Initialization=Kaiming2024.04 | 35.47 | |
| Qwen1.5-1.8Bnumber of parameters=1.8B2024.07 | 28.6 | |
| DDKBase Model=Qwen-1.5 1.8B, Teacher=Qwen-1.5 7B2024.07 | 24.98 | |
| MiniLLMBase Model=Qwen-1.5 1.8B, Teacher=Qwen-1.5 7B2024.07 | 23.31 | |
| KDBase Model=Qwen-1.5 1.8B, Teacher=Qwen-1.5 7B2024.07 | 22.69 | |
| CPTBase Model=Qwen-1.5 1.8B, Teacher=Qwen-1.5 7B2024.07 | 20.45 | |
| Qwen-1.5 1.8BRole=Student2024.07 | 18 | |
| Qwen1.5-0.5Bnumber of parameters=0.5B2024.07 | 14.5 |