Zero-shot Evaluation on ARC-C, BoolQ, HellaSwag, OpenBookQA, PIQA, RTE
55.6ARC-C AccuracyUNPRUNED
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| UNPRUNED#L=-, ACT.=14.2B, MEM.=100%, Speedup=1.0x, SFT=No2024.11 | 55.6 | 88.2 | 82.3 | 45.6 | 82 | 76.9 | 71.8 | |
| CD-MoE-SR#L=6, ACT.=14.2B, MEM.=83.9%, Speedup=1.20x, SFT=No2024.11 | 54.1 | 86.6 | 78.6 | 43 | 80.7 | 80.2 | 70.5 | |
| CD-MoE-S#L=6, ACT.=12.9B, MEM.=81.6%, Speedup=1.22x, SFT=No2024.11 | 54 | 86.7 | 79.1 | 43.4 | 81.5 | 78.5 | 70.5 | |
| MC-SMoE#L=6, ACT.=14.2B, MEM.=76.4%, Speedup=1.0x, SFT=No2024.11 | 53.2 | 87.5 | 77.4 | 42 | 80.2 | 79.4 | 70 | |
| LayerDrop#L=6, ACT.=11.6B, MEM.=79.3%, Speedup=1.24x, SFT=No2024.11 | 42.9 | 77.3 | 69 | 37.6 | 76 | 69.7 | 62.1 | |
| BlockDrop#L=6, ACT.=11.4B, MEM.=79.0%, Speedup=1.35x, SFT=No2024.11 | 28.3 | 61.3 | 51.6 | 32.6 | 62.7 | 59.9 | 49.7 |