Language Modeling on SlimPajama 6B (val)
2.3Validation LossADAMW (DENSE all-reduce)
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| ADAMW (DENSE all-reduce)E2E Time (Days)=2.47 (1x), Architecture=Llama-500M2026.07 | 2.3 | — | — | — | — | — | — | |
| SCAPE (d = 0.1)E2E Time (Days)=1.59 (1.55x), Architecture=Llama-500M2026.07 | 2.3 | — | — | — | — | — | — | |
| SCAPE (d = 0.01)E2E Time (Days)=1.40 (1.76x), Architecture=Llama-500M2026.07 | 2.31 | — | — | — | — | — | — | |
| ADAMS (DENSE all-reduce)E2E Time (Days)=2.47 (1x), Architecture=Llama-500M2026.07 | 2.32 | — | — | — | — | — | — | |
| FP16Model=LLaMA-3.2-1B, Pipeline Parallelism=4, Sequence length=4096, Global batch size=2562025.06 | — | 39.18 | 28.54 | 25.36 | 23.03 | 21.42 | 20.67 | |
| TAH-QuantModel=LLaMA-3.2-1B, Pipeline Parallelism=4, Sequence length=4096, Global batch size=256, Bit allocation=80% INT4 + 20% INT3, Quantization tile size (G)=64, Allocation-tile size (A)=642025.06 | — | 38.47 | 28.51 | 25.3 | 22.99 | 21.41 | 20.64 |