Language Modeling on UL2 Pre-training (val)
1.841LossBaseline
Evaluation Results
| Method | Links | |
|---|---|---|
| BaselineParameters=8B, d(i)/i (block size)=72, Schedule=1x, Speedup=1x2024.09 | 1.841 | |
| MIDASParameters=8B, d(i)/i (block size)=9, Schedule=PROP-2, Speedup=1.26x2024.09 | 1.844 | |
| BaselineParameters=2B, d(i)/i (block size)=48, Schedule=1x, Speedup=1x2024.09 | 1.926 | |
| MIDASParameters=2B, d(i)/i (block size)=8, Schedule=PROP-2, Speedup=1.24x2024.09 | 1.929 | |
| GRADSTACKParameters=2B, d(i)/i (block size)=8, Schedule=PROP-2, Speedup=1.24x2024.09 | 1.945 | |
| MIDASParameters=2B, d(i)/i (block size)=8, Schedule=PROP-1, Speedup=1.39x2024.09 | 1.947 | |
| BaselineParameters=1B, d(i)/i (block size)=24, Schedule=1x, Speedup=1x2024.09 | 1.996 | |
| MIDASParameters=1B, d(i)/i (block size)=4, Schedule=PROP-3, Speedup=1.16x2024.09 | 1.999 | |
| MIDASParameters=1B, d(i)/i (block size)=4, Schedule=PROP-2, Speedup=1.24x2024.09 | 2.009 | |
| MIDASParameters=1B, d(i)/i (block size)=3, Schedule=PROP-2, Speedup=1.26x2024.09 | 2.012 | |
| GRADSTACKParameters=1B, d(i)/i (block size)=4, Schedule=PROP-2, Speedup=1.24x2024.09 | 2.024 | |
| MIDASParameters=1B, d(i)/i (block size)=4, Schedule=PROP-1, Speedup=1.39x2024.09 | 2.028 | |
| MIDASParameters=1B, d(i)/i (block size)=3, Schedule=PROP-1, Speedup=1.41x2024.09 | 2.032 | |
| GRADSTACKParameters=1B, d(i)/i (block size)=4, Schedule=PROP-1, Speedup=1.39x2024.09 | 2.045 |