OpenBook Question Answering on OBQA
44.2AccuracyUNPRUNED
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| UNPRUNEDACT.=2.8B, MEM.=100%, Speedup=1.0x, SFT Protocol=w/o SFT2024.11 | 44.2 | — | |
| CD-MoE-S#L=8, ACT.=2.4B, MEM.=73.1%, Speedup=1.34x, SFT Protocol=w/ lightweight SFT2024.11 | 40.3 | — | |
| OPENLLAMA-3BACT.=3B, SFT Protocol=w/o SFT2024.11 | 38.6 | — | |
| CD-MoE-S#L=8, ACT.=2.4B, MEM.=73.1%, Speedup=1.34x, SFT Protocol=w/o SFT2024.11 | 38.6 | — | |
| CD-MoE-SR#L=9, ACT.=2.8B, MEM.=72.5%, Speedup=1.26x, SFT Protocol=w/ lightweight SFT2024.11 | 38.6 | — | |
| CD-MoE-SR#L=9, ACT.=2.8B, MEM.=72.5%, Speedup=1.26x, SFT Protocol=w/o SFT2024.11 | 37.6 | — | |
| M-SMoE#L=9, ACT.=2.8B, MEM.=70.0%, Speedup=1.0x, SFT Protocol=w/o SFT2024.11 | 36.4 | — | |
| OPT-2.7BACT.=2.7B, SFT Protocol=w/o SFT2024.11 | 35.2 | — | |
| GPT-NEO-2.7BACT.=2.7B, SFT Protocol=w/o SFT2024.11 | 34.6 | — | |
| BlockDrop#L=8, ACT.=2.3B, MEM.=71.3%, Speedup=1.42x, SFT Protocol=w/o SFT2024.11 | 34.6 | — | |
| LayerDrop#L=8, ACT.=2.4B, MEM.=72.2%, Speedup=1.34x, SFT Protocol=w/ lightweight SFT2024.11 | 34.2 | — | |
| LayerDrop#L=8, ACT.=2.4B, MEM.=72.2%, Speedup=1.34x, SFT Protocol=w/o SFT2024.11 | 32.6 | — | |
| BLOOM-3BACT.=3B, SFT Protocol=w/o SFT2024.11 | 32.4 | — | |
| EMO (Stage 5)E=64→1282026.05 | 29.6 | — | |
| FIXED_E=128Experts=1282026.05 | 28.8 | — | |
| FIXED_E=32Experts=322026.05 | 27.8 | — | |
| FIXED_E=16Experts=162026.05 | 27.4 | — | |
| EMO (Stage 4)E=32→642026.05 | 26.6 | — | |
| EMO (Stage 2)E=8→162026.05 | 24.8 | — | |
| EMO (Stage 1)E=82026.05 | 24.2 | — | |
| EMO (Stage 3)E=16→322026.05 | 23 | — | |
| Qwen2.5-Omni-7BSystem Type=End-to-end system2025.10 | 0.855 | 3.5 | |
| ASR + Qwen2.5-7BSystem Type=Cascaded topline, ASR=Whisper-v3-Large2025.10 | 0.84 | 5 | |
| ASR + Qwen2.5-3BSystem Type=Cascaded topline, ASR=Whisper-v3-Large2025.10 | 0.769 | 4.9 | |
| SALAD-3BTraining Stage=Stage II2025.10 | 0.767 | 5.1 | |
| SALAD-7BTraining Stage=Stage II2025.10 | 0.751 | 13.9 | |
| SALAD-7BTraining Stage=Stage I2025.10 | 0.697 | 19.3 | |
| SALAD-3BTraining Stage=Stage I2025.10 | 0.655 | 16.3 | |
| GLM-4-Voice-9BSystem Type=End-to-end system2025.10 | 0.576 | 30.1 | |
| DiVA-Llama3.1-8BSystem Type=End-to-end system2025.10 | 0.409 | 42.4 | |
| Qwen2-Audio-7BSystem Type=End-to-end system2025.10 | 0.396 | 37.1 | |
| Random2025.10 | 0.25 | — |