Sentiment Analysis on SST-2 (Accuracy, Training Loss, Avg)
90.34SST-2 AccuracyMoE on down proj
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MoE on down proj#Expert=8, Parameters=2.35B, FLOPs=919G, MoE position=down proj, Evaluation Protocol=fine-tuned2023.06 | 90.34 | 1.62 | 60.44 | |
| MoE on up proj#Expert=8, Parameters=2.35B, FLOPs=919G, MoE position=up proj, Evaluation Protocol=fine-tuned2023.06 | 90.05 | 1.61 | 60.56 | |
| MoE on up proj#Expert=4, Parameters=1.54B, FLOPs=919G, MoE position=up proj, Evaluation Protocol=fine-tuned2023.06 | 89.67 | 1.72 | 59.79 | |
| MoE on gate#Expert=8, Parameters=2.35B, FLOPs=919G, MoE position=gate, Evaluation Protocol=fine-tuned2023.06 | 89.56 | 1.64 | 60.02 | |
| MoE on down proj#Expert=4, Parameters=1.54B, FLOPs=919G, MoE position=down proj, Evaluation Protocol=fine-tuned2023.06 | 89.13 | 1.7 | 59.61 | |
| MoE on gate#Expert=4, Parameters=1.54B, FLOPs=919G, MoE position=gate, Evaluation Protocol=fine-tuned2023.06 | 88.88 | 1.75 | 59.41 | |
| LLaMA-1B#Expert=none, Parameters=0.94B, FLOPs=919G, MoE position=none, Evaluation Protocol=fine-tuned2023.06 | 88.53 | 1.86 | 58.19 |