Zero-shot Evaluation on 8 Downstream Tasks (ARC-C, BoolQ, HellaSwag, MMLU, OBQA, PIQA, RTE, WinoGrande)
45ARC-C AccuracyUNPRUNED
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| UNPRUNEDACT.=2.7B, MEM.=100%, Speedup=1.0x, Evaluation protocol=zero-shot2024.11 | 45 | 79.5 | 77.3 | 61.2 | 43.6 | 80.3 | 67.9 | 69.3 | 65.5 | |
| CD-MoE-S#L=6, ACT.=2.5B, MEM.=78.3%, Speedup=1.22x, SFT status=W/O SFT, Evaluation protocol=zero-shot2024.11 | 41.1 | 75.8 | 72.9 | 56 | 41 | 78.9 | 67.2 | 68.1 | 62.6 | |
| MC-SMoE#L=6, ACT.=2.7B, MEM.=73.0%, Speedup=1.0x, SFT status=W/O SFT, Evaluation protocol=zero-shot2024.11 | 40.9 | 76.1 | 71.5 | 53.2 | 39.7 | 77.5 | 70.5 | 64.7 | 61.8 | |
| CD-MoE-SR#L=6, ACT.=2.7B, MEM.=79.7%, Speedup=1.19x, SFT status=W/O SFT, Evaluation protocol=zero-shot2024.11 | 40.2 | 77.2 | 72.1 | 54.9 | 39.4 | 78 | 71.1 | 66.4 | 62.4 | |
| LayerDrop#L=6, ACT.=2.3B, MEM.=78.3%, Speedup=1.23x, SFT status=W/O SFT, Evaluation protocol=zero-shot2024.11 | 34.4 | 62.6 | 63.1 | 45.4 | 32.8 | 73.6 | 54.5 | 62.6 | 53.6 | |
| BlockDrop#L=6, ACT.=2.2B, MEM.=76.1%, Speedup=1.36x, SFT status=W/O SFT, Evaluation protocol=zero-shot2024.11 | 31.3 | 62.2 | 57.4 | 38.4 | 32 | 70.7 | 52.7 | 58.2 | 50.4 |