Zero-shot Evaluation Aggregate on ARC-C, BoolQ, HellaSwag, MMLU, OBQA, PIQA, RTE, and Winogrande
61.6Average AccuracyUNPRUNED
Evaluation Results
| Method | Links | |
|---|---|---|
| UNPRUNEDACT.=2.8B, MEM.=100%, Speedup=1.0x, SFT Protocol=w/o SFT2024.11 | 61.6 | |
| CD-MoE-SR#L=9, ACT.=2.8B, MEM.=72.5%, Speedup=1.26x, SFT Protocol=w/ lightweight SFT2024.11 | 60.4 | |
| CD-MoE-S#L=8, ACT.=2.4B, MEM.=73.1%, Speedup=1.34x, SFT Protocol=w/ lightweight SFT2024.11 | 59.1 | |
| CD-MoE-SR#L=9, ACT.=2.8B, MEM.=72.5%, Speedup=1.26x, SFT Protocol=w/o SFT2024.11 | 55.5 | |
| CD-MoE-S#L=8, ACT.=2.4B, MEM.=73.1%, Speedup=1.34x, SFT Protocol=w/o SFT2024.11 | 55.1 | |
| M-SMoE#L=9, ACT.=2.8B, MEM.=70.0%, Speedup=1.0x, SFT Protocol=w/o SFT2024.11 | 54.9 | |
| LayerDrop#L=8, ACT.=2.4B, MEM.=72.2%, Speedup=1.34x, SFT Protocol=w/ lightweight SFT2024.11 | 54.5 | |
| OPENLLAMA-3BACT.=3B, SFT Protocol=w/o SFT2024.11 | 52.7 | |
| BlockDrop#L=8, ACT.=2.3B, MEM.=71.3%, Speedup=1.42x, SFT Protocol=w/o SFT2024.11 | 52.2 | |
| OPT-2.7BACT.=2.7B, SFT Protocol=w/o SFT2024.11 | 50.2 | |
| LayerDrop#L=8, ACT.=2.4B, MEM.=72.2%, Speedup=1.34x, SFT Protocol=w/o SFT2024.11 | 49.4 | |
| GPT-NEO-2.7BACT.=2.7B, SFT Protocol=w/o SFT2024.11 | 48.8 | |
| BLOOM-3BACT.=3B, SFT Protocol=w/o SFT2024.11 | 48.7 |