Large Language Model Training Efficiency on Llama 1.7B 3.2
28.3Energy Reduction (Iso-Time)Kareus
Evaluation results
| Method | Links | ||
|---|---|---|---|
| KareusParallelism Strategy=TP8, Number of Microbatches=16, Sequence Length=4K2026.01 | 28.3 | 26.7 | |
| KareusParallelism Strategy=TP8, Number of Microbatches=8, Sequence Length=4K2026.01 | 26.8 | 27.5 | |
| KareusParallelism Strategy=TP8, Number of Microbatches=8, Sequence Length=8K2026.01 | 23.1 | 23.1 | |
| Nanobatching + PerseusParallelism Strategy=TP8, Number of Microbatches=16, Sequence Length=4K2026.01 | 20.4 | 19.9 | |
| KareusParallelism Strategy=CP2TP4, Number of Microbatches=16, Sequence Length=4K2026.01 | 20.1 | 19.7 | |
| Nanobatching + PerseusParallelism Strategy=TP8, Number of Microbatches=8, Sequence Length=8K2026.01 | 20 | 17.7 | |
| Nanobatching + PerseusParallelism Strategy=TP8, Number of Microbatches=8, Sequence Length=4K2026.01 | 16.8 | 17.8 | |
| KareusParallelism Strategy=CP2TP4, Number of Microbatches=8, Sequence Length=8K2026.01 | 15 | 12.1 | |
| Nanobatching + PerseusParallelism Strategy=CP2TP4, Number of Microbatches=16, Sequence Length=4K2026.01 | 8.9 | 8.4 | |
| KareusParallelism Strategy=CP2TP4, Number of Microbatches=8, Sequence Length=4K2026.01 | 0 | 5.1 | |
| Nanobatching + PerseusParallelism Strategy=CP2TP4, Number of Microbatches=8, Sequence Length=8K2026.01 | -0.7 | -0.4 |