Closed Book Question Answering on TriviaQA, TydiQA, NaturalQuestions, WebQuestions
21.8AccuracyMIDAS
Evaluation Results
| Method | Links | |
|---|---|---|
| MIDASParameters=8B, d(i)/i (block size)=9, Schedule=PROP-2, Speedup=1.26x2024.09 | 21.8 | |
| BaselineParameters=8B, d(i)/i (block size)=72, Schedule=1x, Speedup=1x2024.09 | 21.1 | |
| MIDASParameters=2B, d(i)/i (block size)=8, Schedule=PROP-2, Speedup=1.24x2024.09 | 15.7 | |
| BaselineParameters=2B, d(i)/i (block size)=48, Schedule=1x, Speedup=1x2024.09 | 15.2 | |
| GRADSTACKParameters=2B, d(i)/i (block size)=8, Schedule=PROP-2, Speedup=1.24x2024.09 | 14.2 | |
| MIDASParameters=2B, d(i)/i (block size)=8, Schedule=PROP-1, Speedup=1.39x2024.09 | 14 | |
| BaselineParameters=1B, d(i)/i (block size)=24, Schedule=1x, Speedup=1x2024.09 | 13.2 | |
| MIDASParameters=1B, d(i)/i (block size)=4, Schedule=PROP-3, Speedup=1.16x2024.09 | 12.5 | |
| MIDASParameters=1B, d(i)/i (block size)=3, Schedule=PROP-2, Speedup=1.26x2024.09 | 11.9 | |
| MIDASParameters=1B, d(i)/i (block size)=4, Schedule=PROP-2, Speedup=1.24x2024.09 | 11.7 | |
| MIDASParameters=1B, d(i)/i (block size)=4, Schedule=PROP-1, Speedup=1.39x2024.09 | 11.6 | |
| GRADSTACKParameters=1B, d(i)/i (block size)=4, Schedule=PROP-2, Speedup=1.24x2024.09 | 11 | |
| MIDASParameters=1B, d(i)/i (block size)=3, Schedule=PROP-1, Speedup=1.41x2024.09 | 10.6 | |
| GRADSTACKParameters=1B, d(i)/i (block size)=4, Schedule=PROP-1, Speedup=1.39x2024.09 | 10.3 |