List your company
ResearchFormerly WizWand

Closed Book Question Answering on TriviaQA, TydiQA, NaturalQuestions, WebQuestions

21.8AccuracyMIDAS
9.8412.94516.0519.155Sep 27, 2024
Updated 8mo ago

Evaluation results

MethodLinks
MIDASParameters=8B, d(i)/i (block size)=9, Schedule=PROP-2, Speedup=1.26x2024.0921.8
BaselineParameters=8B, d(i)/i (block size)=72, Schedule=1x, Speedup=1x2024.0921.1
MIDASParameters=2B, d(i)/i (block size)=8, Schedule=PROP-2, Speedup=1.24x2024.0915.7
BaselineParameters=2B, d(i)/i (block size)=48, Schedule=1x, Speedup=1x2024.0915.2
GRADSTACKParameters=2B, d(i)/i (block size)=8, Schedule=PROP-2, Speedup=1.24x2024.0914.2
MIDASParameters=2B, d(i)/i (block size)=8, Schedule=PROP-1, Speedup=1.39x2024.0914
BaselineParameters=1B, d(i)/i (block size)=24, Schedule=1x, Speedup=1x2024.0913.2
MIDASParameters=1B, d(i)/i (block size)=4, Schedule=PROP-3, Speedup=1.16x2024.0912.5
MIDASParameters=1B, d(i)/i (block size)=3, Schedule=PROP-2, Speedup=1.26x2024.0911.9
MIDASParameters=1B, d(i)/i (block size)=4, Schedule=PROP-2, Speedup=1.24x2024.0911.7
MIDASParameters=1B, d(i)/i (block size)=4, Schedule=PROP-1, Speedup=1.39x2024.0911.6
GRADSTACKParameters=1B, d(i)/i (block size)=4, Schedule=PROP-2, Speedup=1.24x2024.0911
MIDASParameters=1B, d(i)/i (block size)=3, Schedule=PROP-1, Speedup=1.41x2024.0910.6
GRADSTACKParameters=1B, d(i)/i (block size)=4, Schedule=PROP-1, Speedup=1.39x2024.0910.3