List your company
ResearchFormerly WizWand

Zero-shot Evaluation Aggregate on ARC-C, BoolQ, HellaSwag, MMLU, OBQA, PIQA, RTE, and Winogrande

61.6Average AccuracyUNPRUNED
48.18451.66755.1558.633Nov 26, 2024
Updated 6mo ago

Evaluation results

MethodLinks
UNPRUNEDACT.=2.8B, MEM.=100%, Speedup=1.0x, SFT Protocol=w/o SFT2024.1161.6
CD-MoE-SR#L=9, ACT.=2.8B, MEM.=72.5%, Speedup=1.26x, SFT Protocol=w/ lightweight SFT2024.1160.4
CD-MoE-S#L=8, ACT.=2.4B, MEM.=73.1%, Speedup=1.34x, SFT Protocol=w/ lightweight SFT2024.1159.1
CD-MoE-SR#L=9, ACT.=2.8B, MEM.=72.5%, Speedup=1.26x, SFT Protocol=w/o SFT2024.1155.5
CD-MoE-S#L=8, ACT.=2.4B, MEM.=73.1%, Speedup=1.34x, SFT Protocol=w/o SFT2024.1155.1
M-SMoE#L=9, ACT.=2.8B, MEM.=70.0%, Speedup=1.0x, SFT Protocol=w/o SFT2024.1154.9
LayerDrop#L=8, ACT.=2.4B, MEM.=72.2%, Speedup=1.34x, SFT Protocol=w/ lightweight SFT2024.1154.5
OPENLLAMA-3BACT.=3B, SFT Protocol=w/o SFT2024.1152.7
BlockDrop#L=8, ACT.=2.3B, MEM.=71.3%, Speedup=1.42x, SFT Protocol=w/o SFT2024.1152.2
OPT-2.7BACT.=2.7B, SFT Protocol=w/o SFT2024.1150.2
LayerDrop#L=8, ACT.=2.4B, MEM.=72.2%, Speedup=1.34x, SFT Protocol=w/o SFT2024.1149.4
GPT-NEO-2.7BACT.=2.7B, SFT Protocol=w/o SFT2024.1148.8
BLOOM-3BACT.=3B, SFT Protocol=w/o SFT2024.1148.7