Language Modeling on SlimPajama-6B (train)
2.1Train LossADAMW (DENSE all-reduce)
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| ADAMW (DENSE all-reduce)E2E Time (Days)=2.47 (1x), Architecture=Llama-500M2026.07 | 2.1 | — | — | — | — | — | — | |
| ADAMS (DENSE all-reduce)E2E Time (Days)=2.47 (1x), Architecture=Llama-500M2026.07 | 2.11 | — | — | — | — | — | — | |
| SCAPE (d = 0.1)E2E Time (Days)=1.59 (1.55x), Architecture=Llama-500M2026.07 | 2.13 | — | — | — | — | — | — | |
| SCAPE (d = 0.01)E2E Time (Days)=1.40 (1.76x), Architecture=Llama-500M2026.07 | 2.17 | — | — | — | — | — | — | |
| FP16Model=LLaMA-3.2-1B, Pipeline Parallelism=4, Sequence length=4096, Global batch size=2562025.06 | — | 3.778 | 3.326 | 3.149 | 3.032 | 2.954 | 2.893 | |
| TAH-QuantModel=LLaMA-3.2-1B, Pipeline Parallelism=4, Sequence length=4096, Global batch size=256, Bit allocation=80% INT4 + 20% INT3, Quantization tile size (G)=64, Allocation-tile size (A)=642025.06 | — | 3.781 | 3.325 | 3.147 | 3.031 | 2.953 | 2.891 |