List your company
ResearchFormerly WizWand

Language Modeling on UL2 Pre-training (val)

1.841LossBaseline
1.832841.887921.9431.99808Sep 27, 2024
Updated 8mo ago

Evaluation results

MethodLinks
BaselineParameters=8B, d(i)/i (block size)=72, Schedule=1x, Speedup=1x2024.091.841
MIDASParameters=8B, d(i)/i (block size)=9, Schedule=PROP-2, Speedup=1.26x2024.091.844
BaselineParameters=2B, d(i)/i (block size)=48, Schedule=1x, Speedup=1x2024.091.926
MIDASParameters=2B, d(i)/i (block size)=8, Schedule=PROP-2, Speedup=1.24x2024.091.929
GRADSTACKParameters=2B, d(i)/i (block size)=8, Schedule=PROP-2, Speedup=1.24x2024.091.945
MIDASParameters=2B, d(i)/i (block size)=8, Schedule=PROP-1, Speedup=1.39x2024.091.947
BaselineParameters=1B, d(i)/i (block size)=24, Schedule=1x, Speedup=1x2024.091.996
MIDASParameters=1B, d(i)/i (block size)=4, Schedule=PROP-3, Speedup=1.16x2024.091.999
MIDASParameters=1B, d(i)/i (block size)=4, Schedule=PROP-2, Speedup=1.24x2024.092.009
MIDASParameters=1B, d(i)/i (block size)=3, Schedule=PROP-2, Speedup=1.26x2024.092.012
GRADSTACKParameters=1B, d(i)/i (block size)=4, Schedule=PROP-2, Speedup=1.24x2024.092.024
MIDASParameters=1B, d(i)/i (block size)=4, Schedule=PROP-1, Speedup=1.39x2024.092.028
MIDASParameters=1B, d(i)/i (block size)=3, Schedule=PROP-1, Speedup=1.41x2024.092.032
GRADSTACKParameters=1B, d(i)/i (block size)=4, Schedule=PROP-1, Speedup=1.39x2024.092.045