Stochastic Optimization Convergence on Theoretical Analysis
4Convergence Rate BoundNormalized SGD
Evaluation results
| Method | Links | |
|---|---|---|
| Normalized SGDClass=Stochastic LMO, Gradient Evaluations=1, LMO set C=|| · ||2–ball, LMOC(g)=-g/||g||2026.05 | 4 | |
| signSGDClass=Stochastic LMO, Gradient Evaluations=1, LMO set C=|| · ||∞–ball, LMOC(g)=−sign(g)2026.05 | 4 | |
| SignumClass=Stochastic LMO, Gradient Evaluations=1, LMO set C=|| · ||∞–ball, LMOC(g)=−sign(g)2026.05 | 4 | |
| LionClass=Stochastic LMO, Gradient Evaluations=1, LMO set C=|| · ||∞–ball, LMOC(g)=−sign(g)2026.05 | 4 | |
| MuonClass=Stochastic LMO, Gradient Evaluations=1, LMO set C=|| · ||op–ball*, LMOC(g)=−UV⊤2026.05 | 4 | |
| NIGTClass=LMO-IGT, Gradient Evaluations=1, LMO set C=|| · ||2–ball, LMOC(g)=−g/||g||2026.05 | 3.5 | |
| Lion-IGTClass=LMO-IGT, Gradient Evaluations=1, LMO set C=|| · ||∞–ball, LMOC(g)=−sign(g)2026.05 | 3.5 | |
| Muon-IGTClass=LMO-IGT, Gradient Evaluations=1, LMO set C=|| · ||op–ball*, LMOC(g)=−UV⊤2026.05 | 3.5 | |
| Lion-VRClass=LMO-VR, Gradient Evaluations=2, LMO set C=|| · ||∞–ball, LMOC(g)=−sign(g)2026.05 | 3 | |
| MARS-ShampooClass=LMO-VR, Gradient Evaluations=2, LMO set C=|| · ||op–ball*, LMOC(g)=−UV⊤2026.05 | 3 | |
| Muon-VRClass=LMO-VR, Gradient Evaluations=2, LMO set C=|| · ||op–ball*, LMOC(g)=−UV⊤2026.05 | 3 | |
| Qian et al.Learning rate regime (η)=2/3 < η < 12026.05 | 3 | |
| Poisson-Moreau DriftLearning rate regime (η)=1/2 < η ≤ 2/32026.05 | 2 | |
| Poisson-Moreau DriftLearning rate regime (η)=2/3 < η < 12026.05 | 2 | |
| D-PSGDCommunication=none, Assumptions=3, 5, 112026.03 | 1 | |
| CHOCO-SGDCommunication=A, Assumptions=3, 4, 5, 10, 112026.03 | 1 | |
| SQuARM-SGDCommunication=A, Assumptions=3, 4, 5, 7, 10, 112026.03 | 1 | |
| DFedSAMCommunication=B, Assumptions=3, 4, 5, 112026.03 | 1 | |
| ELCommunication=B, Assumptions=3, 4, 5, 72026.03 | 1 | |
| DCD-PSGDCommunication=A, Assumptions=3, 4, 5, 8, 112026.03 | 1 | |
| BEERCommunication=A, Assumptions=3, 5, 7, 10, 112026.03 | 1 | |
| Qian et al.Learning rate regime (η)=η = 12026.05 | 1 | |
| Poisson-Moreau DriftLearning rate regime (η)=η = 12026.05 | -1 |