Language Modeling on 200k tokens seq_len=2048
9.34PPLexactNanochat-3B Baseline
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Nanochat-3B BaselineType=3B, Best SNLP=n8-K4, Boots=46, NFE=282026.07 | 9.34 | 1.246 | 1.212 | 3.5 | |
| Nanochat-3B IDN Reg.Type=3B, Best SNLP=n8-K4, Boots=46, NFE=282026.07 | 9.34 | 1.27 | 1.267 | 0.3 | |
| Nanochat-0.5B-mHC BaselineType=0.5B mHC, Best SNLP=n8-K4, Boots=46, NFE=282026.07 | 13.83 | 1.309 | 1.292 | 1.7 | |
| Nanochat-0.5B standard BaselineType=0.5B, Best SNLP=n8-K4, Boots=46, NFE=282026.07 | 13.91 | 1.472 | 1.462 | 0.9 | |
| Nanochat-0.5B standard IDN Reg.Type=0.5B, Best SNLP=n24-K4, Boots=20, NFE=122026.07 | 14.07 | 1.419 | 1.355 | 6.4 | |
| Nanochat-0.5B-mHC HCN Reg.Type=0.5B mHC, Best SNLP=n20-K4, Boots=26, NFE=162026.07 | 14.19 | 1.238 | 1.217 | 2.1 | |
| Nanochat-0.5B w/o x0/VE BaselineType=0.5B, Best SNLP=n8-K4, Boots=46, NFE=282026.07 | 15.89 | 1.466 | 1.457 | 0.9 | |
| Nanochat-0.5B w/o x0/VE IDN Reg.Type=0.5B, Best SNLP=n24-K4, Boots=20, NFE=122026.07 | 15.95 | 1.524 | 1.499 | 2.5 |