Language Modeling on C4 (train)
15.28PPL8-bit COAP
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 8-bit COAPModel=LLaMA-7B (80K), Optimizer Mem. (GB)=5.25 (-58%), Model Mem. (GB)=12.55, Training Time=-2%2024.11 | 15.28 | — | — | — | — | — | — | — | — | |
| 8-bit AdamModel=LLaMA-7B (80K), Optimizer Mem. (GB)=12.55, Model Mem. (GB)=12.55, Training Time=52.01 h2024.11 | 15.39 | — | — | — | — | — | — | — | — | |
| 8-bit GaLoreModel=LLaMA-7B (80K), Optimizer Mem. (GB)=5.25 (-58%), Model Mem. (GB)=12.55, Training Time=+19%2024.11 | 15.47 | — | — | — | — | — | — | — | — | |
| LOROModel Size=1B, r/d=512/2048, Tokens=13.1B, Param(M)=609, Mem(G)=3.662026.05 | 15.53 | — | — | — | — | — | — | — | — | |
| AdamWModel=LLaMA-1B (100K), Optimizer Mem. (GB)=4.99, Model Mem. (GB)=2.49, Training Time=28.50 h2024.11 | 15.56 | — | — | — | — | — | — | — | — | |
| COAPModel=LLaMA-1B (100K), Optimizer Mem. (GB)=1.94 (-61%), Model Mem. (GB)=2.49, Training Time=+2%2024.11 | 15.56 | — | — | — | — | — | — | — | — | |
| Full-RankModel Size=1B, r/d=512/2048, Tokens=13.1B, Param(M)=1339, Mem(G)=8.042026.05 | 15.56 | — | — | — | — | — | — | — | — | |
| GaLoreModel=LLaMA-1B (100K), Optimizer Mem. (GB)=1.94 (-61%), Model Mem. (GB)=2.49, Training Time=+17%2024.11 | 15.64 | — | — | — | — | — | — | — | — | |
| GaLoreModel Size=1B, r/d=512/2048, Tokens=13.1B, Param(M)=1339, Mem(G)=4.762026.05 | 15.64 | — | — | — | — | — | — | — | — | |
| ELASModel Size=1B, r/d=512/2048, Tokens=13.1B, Param(M)=609, Mem(G)=3.662026.05 | 15.69 | — | — | — | — | — | — | — | — | |
| CoLAModel Size=1B, r/d=512/2048, Tokens=13.1B, Param(M)=609, Mem(G)=3.662026.05 | 15.76 | — | — | — | — | — | — | — | — | |
| SLTrainModel Size=1B, r/d=512/2048, Tokens=13.1B, Param(M)=646, Mem(G)=4.162026.05 | 16.14 | — | — | — | — | — | — | — | — | |
| ReLORAModel=LLaMA-1B (100K), Optimizer Mem. (GB)=2.27 (-55%), Model Mem. (GB)=3.38 (+36%), Training Time=+6%2024.11 | 18.33 | — | — | — | — | — | — | — | — | |
| ReLoRAModel Size=1B, r/d=512/2048, Tokens=13.1B, Param(M)=1339, Mem(G)=6.342026.05 | 18.33 | — | — | — | — | — | — | — | — | |
| Full-RankModel Size=350M, r/d=256/1024, Tokens=6.4B, Param(M)=368, Mem(G)=2.212026.05 | 18.8 | — | — | — | — | — | — | — | — | |
| GaLoreModel Size=350M, r/d=256/1024, Tokens=6.4B, Param(M)=368, Mem(G)=1.592026.05 | 18.95 | — | — | — | — | — | — | — | — | |
| LoRAModel=LLaMA-1B (100K), Optimizer Mem. (GB)=2.27 (-55%), Model Mem. (GB)=3.38 (+36%), Training Time=+6%2024.11 | 19.21 | — | — | — | — | — | — | — | — | |
| LoRAModel Size=1B, r/d=512/2048, Tokens=13.1B, Param(M)=1339, Mem(G)=6.342026.05 | 19.21 | — | — | — | — | — | — | — | — | |
| SLTrainModel Size=350M, r/d=256/1024, Tokens=6.4B, Param(M)=194, Mem(G)=1.242026.05 | 19.42 | — | — | — | — | — | — | — | — | |
| LOROModel Size=350M, r/d=256/1024, Tokens=6.4B, Param(M)=185, Mem(G)=1.112026.05 | 19.66 | — | — | — | — | — | — | — | — | |
| CoLAModel Size=350M, r/d=256/1024, Tokens=6.4B, Param(M)=185, Mem(G)=1.112026.05 | 19.75 | — | — | — | — | — | — | — | — | |
| ELASModel Size=350M, r/d=256/1024, Tokens=6.4B, Param(M)=185, Mem(G)=1.112026.05 | 19.94 | — | — | — | — | — | — | — | — | |
| HTMuonModel Scale=LLaMA-135M, Hardware=4 NVIDIA RTX PRO 6000 GPUs2026.03 | 21.24 | — | 6.57 | — | — | — | — | — | — | |
| HTMuon_NSModel Scale=LLaMA-135M, Shuffle=False, Hardware=4 NVIDIA RTX PRO 6000 GPUs2026.03 | 21.36 | — | 2.17 | — | — | — | — | — | — | |
| HTMuon_NSModel Scale=LLaMA-135M, Update Interval=5, Shuffle=False, Hardware=4 NVIDIA RTX PRO 6000 GPUs2026.03 | 22.02 | — | 1.55 | — | — | — | — | — | — | |
| HTMuonModel Scale=LLaMA-135M, Update Interval=5, Hardware=4 NVIDIA RTX PRO 6000 GPUs2026.03 | 22.04 | — | 2.4 | — | — | — | — | — | — | |
| MuonModel Scale=LLaMA-135M, Hardware=4 NVIDIA RTX PRO 6000 GPUs2026.03 | 22.27 | — | 1.4 | — | — | — | — | — | — | |
| Full-RankModel Size=130M, r/d=256/768, Tokens=2.2B, Param(M)=134, Mem(G)=0.812026.05 | 24.36 | — | — | — | — | — | — | — | — | |
| LOROModel Size=130M, r/d=256/768, Tokens=2.2B, Param(M)=94, Mem(G)=0.572026.05 | 24.78 | — | — | — | — | — | — | — | — | |
| ELASModel Size=130M, r/d=256/768, Tokens=2.2B, Param(M)=94, Mem(G)=0.572026.05 | 24.8 | — | — | — | — | — | — | — | — | |
| GaLoreModel Size=130M, r/d=256/768, Tokens=2.2B, Param(M)=134, Mem(G)=0.612026.05 | 25.36 | — | — | — | — | — | — | — | — | |
| LoRAModel Size=350M, r/d=256/1024, Tokens=6.4B, Param(M)=368, Mem(G)=1.852026.05 | 25.58 | — | — | — | — | — | — | — | — | |
| CoLAModel Size=130M, r/d=256/768, Tokens=2.2B, Param(M)=94, Mem(G)=0.572026.05 | 25.61 | — | — | — | — | — | — | — | — | |
| SLTrainModel Size=130M, r/d=256/768, Tokens=2.2B, Param(M)=97, Mem(G)=0.602026.05 | 26.04 | — | — | — | — | — | — | — | — | |
| HTMuonModel Scale=LLaMA-60M, Hardware=4 NVIDIA RTX PRO 6000 GPUs2026.03 | 27.87 | 2.46 | — | — | — | — | — | — | — | |
| HTMuon_NSModel Scale=LLaMA-60M, Shuffle=False, Hardware=4 NVIDIA RTX PRO 6000 GPUs2026.03 | 28.06 | 0.87 | — | — | — | — | — | — | — | |
| HTMuonModel Scale=LLaMA-60M, Update Interval=5, Hardware=4 NVIDIA RTX PRO 6000 GPUs2026.03 | 28.36 | 1.02 | — | — | — | — | — | — | — | |
| HTMuon_NSModel Scale=LLaMA-60M, Update Interval=5, Shuffle=False, Hardware=4 NVIDIA RTX PRO 6000 GPUs2026.03 | 28.5 | 0.71 | — | — | — | — | — | — | — | |
| MuonModel Scale=LLaMA-60M, Hardware=4 NVIDIA RTX PRO 6000 GPUs2026.03 | 28.84 | 0.67 | — | — | — | — | — | — | — | |
| ReLoRAModel Size=350M, r/d=256/1024, Tokens=6.4B, Param(M)=368, Mem(G)=1.852026.05 | 29.08 | — | — | — | — | — | — | — | — | |
| ReLoRAModel Size=130M, r/d=256/768, Tokens=2.2B, Param(M)=134, Mem(G)=0.842026.05 | 29.37 | — | — | — | — | — | — | — | — | |
| LOROModel Size=60M, r/d=128/512, Tokens=1.1B, Param(M)=43, Mem(G)=0.242026.05 | 33.87 | — | — | — | — | — | — | — | — | |
| LoRAModel Size=130M, r/d=256/768, Tokens=2.2B, Param(M)=134, Mem(G)=0.842026.05 | 33.92 | — | — | — | — | — | — | — | — | |
| Full-RankModel Size=60M, r/d=128/512, Tokens=1.1B, Param(M)=58, Mem(G)=0.352026.05 | 34.06 | — | — | — | — | — | — | — | — | |
| CoLAModel Size=60M, r/d=128/512, Tokens=1.1B, Param(M)=43, Mem(G)=0.242026.05 | 34.1 | — | — | — | — | — | — | — | — | |
| ELASModel Size=60M, r/d=128/512, Tokens=1.1B, Param(M)=43, Mem(G)=0.242026.05 | 34.12 | — | — | — | — | — | — | — | — | |
| SLTrainModel Size=60M, r/d=128/512, Tokens=1.1B, Param(M)=44, Mem(G)=0.262026.05 | 34.15 | — | — | — | — | — | — | — | — | |
| GaLoreModel Size=60M, r/d=128/512, Tokens=1.1B, Param(M)=58, Mem(G)=0.282026.05 | 34.88 | — | — | — | — | — | — | — | — | |
| LoRAModel Size=60M, r/d=128/512, Tokens=1.1B, Param(M)=58, Mem(G)=0.362026.05 | 34.99 | — | — | — | — | — | — | — | — | |
| ReLoRAModel Size=60M, r/d=128/512, Tokens=1.1B, Param(M)=58, Mem(G)=0.362026.05 | 37.04 | — | — | — | — | — | — | — | — | |
| APOLLOMemory Consumption (G)=16.14G, Model (LLaMA 7B)=LLaMA 7B2025.06 | — | — | — | — | — | 17.55 | 14.39 | 13.23 | 13.02 | |
| APOLLO-MiniMemory Consumption (G)=14.53G, Model (LLaMA 7B)=LLaMA 7B2025.06 | — | — | — | — | — | 18.03 | 14.6 | 13.32 | 13.09 | |
| Centralized baselineBandwidth=10 Gbps, Compression=1×2026.04 | — | — | — | 7,530 | 3.86 | — | — | — | — | |
| Decentralized baselineBandwidth=80 Mbps, Compression=1×2026.04 | — | — | — | 609 | 5.28 | — | — | — | — | |
| MuonMemory Consumption (G)=26.95G, Model (LLaMA 7B)=LLaMA 7B, Hardware (8xNVIDIA H200 141G GPUs)=8xNVIDIA H200 141G GPUs2025.06 | — | — | — | — | — | 16.43 | 13.95 | 12.85 | 12.72 | |
| ResBM AdamW (100×)Bandwidth=80 Mbps, Compression=100×, Optimizer=AdamW2026.04 | — | — | — | 7,681 | 4.1 | — | — | — | — | |
| ResBM AdamW (128×)Bandwidth=80 Mbps, Compression=128×, Optimizer=AdamW2026.04 | — | — | — | 7,837 | 4.05 | — | — | — | — | |
| ResBM Muon (100×)Bandwidth=80 Mbps, Compression=100×, Optimizer=Muon2026.04 | — | — | — | 6,795 | 4.03 | — | — | — | — | |
| ResBM Muon (128×)Bandwidth=80 Mbps, Compression=128×, Optimizer=Muon2026.04 | — | — | — | 6,795 | 3.98 | — | — | — | — | |
| SCALEMemory Consumption (G)=13.74G, Model (LLaMA 7B)=LLaMA 7B, Hardware (8xNVIDIA H200 141G GPUs)=8xNVIDIA H200 141G GPUs2025.06 | — | — | — | — | — | 17.99 | 14.57 | 12.86 | 12.59 |