LLM Pretraining on PleIAs SYNTH (val)
0.7578BPB (Validation)ALPHALAB + Opus 4.6
Evaluation results
| Method | Links | |
|---|---|---|
| ALPHALAB + Opus 4.6Best config=10L×752d, QK-norm, Cost=∼$2002026.03 | 0.7578 | |
| ALPHALAB + Sonnet 4.6Best config=11L×768d, QK-norm, Muon, Cost=∼$1202026.03 | 0.8686 | |
| ALPHALAB + GPT-5.2Best config=8L×512d, GQA, cosine, Cost=∼$1502026.03 | 0.9697 | |
| Greedy loop (GPT-5.2)Best config=12L×768d, LLaMA, AdamW, Cost=∼$502026.03 | 1.02 | |
| Single-shot (GPT-5.2)Best config=27.4M LLaMA-style, Cost=<$12026.03 | 1.248 |