Language Modeling on Tülü 3 SFT Mixture
9.96Perplexity (PPL)4-vium
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| 4-viumDepth=48, Hidden width=768, beta=0.152026.05 | 9.96 | 18.45 | |
| 4-viumDepth=24, Hidden width=768, beta=0.152026.05 | 10.91 | 21.73 | |
| 4-viumDepth=36, Hidden width=768, beta=0.152026.05 | 11.47 | 9.15 | |
| BaselineDepth=48, Hidden width=7682026.05 | 12.22 | — | |
| BaselineDepth=36, Hidden width=7682026.05 | 12.63 | — | |
| 4-viumDepth=12, Hidden width=768, beta=0.152026.05 | 13.93 | 14.78 | |
| BaselineDepth=24, Hidden width=7682026.05 | 13.94 | — | |
| BaselineDepth=12, Hidden width=7682026.05 | 16.35 | — | |
| 4-viumDepth=6, Hidden width=768, beta=0.152026.05 | 17.37 | 21.34 | |
| BaselineDepth=6, Hidden width=7682026.05 | 22.09 | — |