List your company
ResearchFormerly WizWand

OpenBook Question Answering on OBQA

44.2AccuracyUNPRUNED
-1.50810.358522.22534.0915Nov 26, 2024Feb 22, 2025May 22, 2025Aug 19, 2025Nov 16, 2025Feb 13, 2026May 13, 2026
Updated 5mo ago

Evaluation results

MethodLinks
UNPRUNEDACT.=2.8B, MEM.=100%, Speedup=1.0x, SFT Protocol=w/o SFT2024.1144.2—
CD-MoE-S#L=8, ACT.=2.4B, MEM.=73.1%, Speedup=1.34x, SFT Protocol=w/ lightweight SFT2024.1140.3—
OPENLLAMA-3BACT.=3B, SFT Protocol=w/o SFT2024.1138.6—
CD-MoE-S#L=8, ACT.=2.4B, MEM.=73.1%, Speedup=1.34x, SFT Protocol=w/o SFT2024.1138.6—
CD-MoE-SR#L=9, ACT.=2.8B, MEM.=72.5%, Speedup=1.26x, SFT Protocol=w/ lightweight SFT2024.1138.6—
CD-MoE-SR#L=9, ACT.=2.8B, MEM.=72.5%, Speedup=1.26x, SFT Protocol=w/o SFT2024.1137.6—
M-SMoE#L=9, ACT.=2.8B, MEM.=70.0%, Speedup=1.0x, SFT Protocol=w/o SFT2024.1136.4—
OPT-2.7BACT.=2.7B, SFT Protocol=w/o SFT2024.1135.2—
GPT-NEO-2.7BACT.=2.7B, SFT Protocol=w/o SFT2024.1134.6—
BlockDrop#L=8, ACT.=2.3B, MEM.=71.3%, Speedup=1.42x, SFT Protocol=w/o SFT2024.1134.6—
LayerDrop#L=8, ACT.=2.4B, MEM.=72.2%, Speedup=1.34x, SFT Protocol=w/ lightweight SFT2024.1134.2—
LayerDrop#L=8, ACT.=2.4B, MEM.=72.2%, Speedup=1.34x, SFT Protocol=w/o SFT2024.1132.6—
BLOOM-3BACT.=3B, SFT Protocol=w/o SFT2024.1132.4—
EMO (Stage 5)E=64→1282026.0529.6—
FIXED_E=128Experts=1282026.0528.8—
FIXED_E=32Experts=322026.0527.8—
FIXED_E=16Experts=162026.0527.4—
EMO (Stage 4)E=32→642026.0526.6—
EMO (Stage 2)E=8→162026.0524.8—
EMO (Stage 1)E=82026.0524.2—
EMO (Stage 3)E=16→322026.0523—
Qwen2.5-Omni-7BSystem Type=End-to-end system2025.100.8553.5
ASR + Qwen2.5-7BSystem Type=Cascaded topline, ASR=Whisper-v3-Large2025.100.845
ASR + Qwen2.5-3BSystem Type=Cascaded topline, ASR=Whisper-v3-Large2025.100.7694.9
SALAD-3BTraining Stage=Stage II2025.100.7675.1
SALAD-7BTraining Stage=Stage II2025.100.75113.9
SALAD-7BTraining Stage=Stage I2025.100.69719.3
SALAD-3BTraining Stage=Stage I2025.100.65516.3
GLM-4-Voice-9BSystem Type=End-to-end system2025.100.57630.1
DiVA-Llama3.1-8BSystem Type=End-to-end system2025.100.40942.4
Qwen2-Audio-7BSystem Type=End-to-end system2025.100.39637.1
Random2025.100.25—