Optimization Hyper-parameter Laws for Large Language Models

About

Large Language Models have driven significant AI advancements, yet their training is resource-intensive and highly sensitive to hyper-parameter selection. While scaling laws provide valuable guidance on model size and data requirements, they fall short in choosing dynamic hyper-parameters, such as learning-rate (LR) schedules, that evolve during training. To bridge this gap, we present Optimization Hyper-parameter Laws (Opt-Laws), a framework that predicts final training loss as a function of LR schedule, model size, and data size. Grounded in SDE-based convergence and escape analyses, Opt-Laws yield interpretable convergence and escape features that predict final training loss across model scales, enabling schedule pre-selection from small-scale experiments. Empirically, Opt-Laws achieve a 94% Top-2 hit rate for identifying near-optimal schedule candidates on held-out configurations, correctly identify the best-performing schedule family in all five evaluated out-of-family settings, and detect training divergence with F1 = 0.92.

Xingyu Xie, Kuangyu Ding, Shuicheng Yan, Kim-Chuan Toh, Tianwen Wei · 2024

Related benchmarks

Task	Dataset	Result
Loss prediction and hyper-parameter configuration ranking	LLM training loss grids 5-fold configuration-level cross-validation (held-out)	R20.998	6
Continual Training	100B English tokens Weak distribution shift (near-tie)	Observed Loss—	5
Divergence prediction	Divergence prediction benchmark 234 convergent vs. 26 divergent configurations (5-fold CV)	Precision96	4
Pre-training extrapolation	configurations 300B single-phase Table 2	Mean Relative Error0.1	4
Pre-training extrapolation	Table 2 configurations 100B multi-phase	Mean Relative Error0.2	4
Pre-training extrapolation	configurations 100B loss-equivalent Table 2	Mean Relative Error0.2	4
Pre-training extrapolation	Table 2 configurations All 10	Mean Relative Error0.2	4
Continual Training	100B Chinese tokens Strong distribution shift	Observed Loss—	3
Fine-tuning ranking evaluation	English fine-tuning corpus 3B tokens	Observed Loss—	3
Fine-tuning ranking evaluation	English fine-tuning corpus 30B tokens	Observed Loss—	3