Language Modeling on ClimbMix FineWeb-Edu 1M-token (val)
30.12PPLIDN Reg.
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| IDN Reg.Model Architecture Variant=Nanochat-3B standard, Sequential PPL Baseline=35.31, Regularization Gain=-5.0%, Config=12xF1-h0, K=82026.05 | 30.12 | -14.7 | 0.57 | 79.9 | 99.3 | 95 | 95.5 | |
| DiagN Reg.Model Architecture Variant=Nanochat-3B standard, Sequential PPL Baseline=35.41, Regularization Gain=-4.7%, Config=12xF1-h0, K=82026.05 | 31.39 | -11.3 | 0.57 | 79.2 | 99 | 90.7 | 87.7 | |
| No Reg.Model Architecture Variant=Nanochat-3B standard, Sequential PPL Baseline=37.16, Config=12xF1-h0, K=82026.05 | 31.45 | -15.4 | 0.59 | 80.3 | 98.9 | 94.3 | 92.2 | |
| DiagN Reg.Model Architecture Variant=Nanochat-3B standard, Sequential PPL Baseline=35.41, Regularization Gain=-4.7%, Config=4xF1-fwd, K=22026.05 | 31.52 | -11 | 0.92 | 87.7 | 98.9 | 94.8 | 68.8 | |
| No Reg.Model Architecture Variant=Nanochat-3B standard, Sequential PPL Baseline=37.16, Config=4xF1-fwd, K=22026.05 | 32.27 | -13.2 | 0.94 | 87.2 | 99.2 | 95.5 | 78.3 | |
| IDN Reg.Model Architecture Variant=Nanochat-3B standard, Sequential PPL Baseline=35.31, Regularization Gain=-5.0%, Config=4xF1-fwd, K=22026.05 | 32.96 | -6.7 | 0.99 | 90.5 | 99.8 | 96.9 | 91 | |
| IDN Reg.Model Architecture Variant=Nanochat-0.5B standard, Sequential PPL Baseline=53.25, Regularization Gain=-23.4%, Config=2xF6-fwd, K=12026.05 | 44 | -17.4 | 1.37 | 68.9 | 98.8 | 93.5 | 98.8 | |
| No Reg.Model Architecture Variant=Nanochat-0.5B standard, Sequential PPL Baseline=69.54, Config=16xF1-h0, K=82026.05 | 47.25 | -32.1 | 0.78 | 50.9 | 97.7 | 88.9 | 72.3 | |
| DiagN Reg.Model Architecture Variant=Nanochat-0.5B standard, Sequential PPL Baseline=63.08, Regularization Gain=-9.3%, Config=12xF1-h0, K=82026.05 | 51.42 | -18.5 | 0.88 | 64.9 | 98.4 | 94.3 | 90.8 | |
| IDN Reg.Model Architecture Variant=Nanochat-0.5B standard, Sequential PPL Baseline=53.25, Regularization Gain=-23.4%, Config=12xF2-h0, K=22026.05 | 53.68 | 0.8 | 2.37 | 53.2 | 97.9 | 87.9 | 35.3 | |
| No Reg.Model Architecture Variant=Nanochat-0.5B-mHC, Sequential PPL Baseline=73.24, Config=4xF2-fwd, K=22026.05 | 61.34 | -16.2 | 1.13 | 66.7 | 99 | 95.5 | 89.2 | |
| No Reg.Model Architecture Variant=Nanochat-0.5B standard, Sequential PPL Baseline=69.54, Config=12xF1-h0, K=42026.05 | 62.01 | -10.8 | 1.11 | 62.6 | 98.2 | 95.2 | 61.4 | |
| DiagN Reg.Model Architecture Variant=Nanochat-0.5B standard, Sequential PPL Baseline=63.08, Regularization Gain=-9.3%, Config=4xF2-h0, K=22026.05 | 63.4 | 0.5 | 1.17 | 61 | 97.9 | 93.8 | 50.4 | |
| HCN Reg.Model Architecture Variant=Nanochat-0.5B-mHC, Sequential PPL Baseline=67.23, Regularization Gain=-8.2%, Config=8xF1-h0, K=12026.05 | 65.91 | -2 | 0.97 | 91.1 | 99.8 | 98.8 | 82.8 | |
| HCN Reg.Model Architecture Variant=Nanochat-0.5B-mHC, Sequential PPL Baseline=67.23, Regularization Gain=-8.2%, Config=20xF1-h0, K=42026.05 | 66.56 | -1 | 1.22 | 91.1 | 99.3 | 98.6 | 80.1 | |
| No Reg.Model Architecture Variant=Nanochat-0.5B-mHC, Sequential PPL Baseline=73.24, Config=4xF3-fwd, K=22026.05 | 69.42 | -5.2 | 1.31 | 56.1 | 98 | 93.7 | 71.1 | |
| IDN Reg.Model Architecture Variant=Nanochat-0.5B w/o x0ve, Sequential PPL Baseline=79.96, Regularization Gain=-5.6%, Config=4xF4-h0, K=42026.05 | 72.71 | -9.1 | 1.24 | 74 | 99.6 | 95.2 | 95.9 | |
| IDN Reg.Model Architecture Variant=Nanochat-0.5B w/o x0ve, Sequential PPL Baseline=79.96, Regularization Gain=-5.6%, Config=4xF6-h0, K=22026.05 | 75.09 | -6.1 | 2.32 | 73.6 | 99.6 | 95.2 | 84.5 | |
| No Reg.Model Architecture Variant=Nanochat-0.5B w/o x0ve, Sequential PPL Baseline=84.74, Config=6xF2-fwd, K=42026.05 | 78.54 | -7.3 | 1.05 | 90.7 | 99.8 | 98.6 | 97.9 | |
| No Reg.Model Architecture Variant=Nanochat-0.5B w/o x0ve, Sequential PPL Baseline=84.74, Config=8xF2-fwd, K=22026.05 | 81.35 | -4 | 1.39 | 76.1 | 99.2 | 96.5 | 55.3 |