Language Modeling on OpenWebText 100k steps (val)
3.004Validation Loss (100k steps)Query weight elimination (larger MLP)
Evaluation results
| Method | Links | |
|---|---|---|
| Query weight elimination (larger MLP)Trainable W_Q=false, Total params=124.37M, Non-emb params=84.95M, MLP hid. mult.=4.5×, Attention scale=1/(2√64), Weight decay=2⁻⁵2025.10 | 3.004 | |
| Baseline (124M)Trainable W_Q=true, Total params=124.37M, Non-emb params=84.95M, MLP hid. mult.=4×, Attention scale=1/√64, Weight decay=0.12025.10 | 3.016 | |
| Query weight elimination (117M)Trainable W_Q=false, Total params=117.30M, Non-emb params=77.88M, MLP hid. mult.=4×, Attention scale=1/(2√64), Weight decay=2⁻⁵2025.10 | 3.018 | |
| Baseline (smaller MLP)Trainable W_Q=true, Total params=117.30M, Non-emb params=77.88M, MLP hid. mult.=3.5×, Attention scale=1/√64, Weight decay=0.12025.10 | 3.026 | |
| Baseline (smaller d)Trainable W_Q=true, Total params=117.92M, Non-emb params=79.73M, MLP hid. mult.=4×, Attention scale=1/√62, Weight decay=0.12025.10 | 3.027 |