LLM Training Optimization on Llama 3.2 3B
12.3Training Time Reduction (%)Kareus
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| KareusParallelism=TP8, µBatch Size=8, Sequence Length=4K2026.01 | 12.3 | 19.6 | — | — | |
| Nanobatching + PerseusParallelism=TP8, µBatch Size=8, Sequence Length=4K2026.01 | 8.5 | 15.5 | — | — | |
| KareusParallelism=CP2TP4, µBatch Size=16, Sequence Length=4K2026.01 | 8 | 16.2 | — | — | |
| KareusParallelism=CP2TP4, µBatch Size=8, Sequence Length=8K2026.01 | 6.2 | 14.9 | — | — | |
| KareusParallelism=CP2TP4, µBatch Size=8, Sequence Length=4K2026.01 | 5.2 | 14.4 | — | — | |
| Nanobatching + PerseusParallelism=CP2TP4, µBatch Size=16, Sequence Length=4K2026.01 | 2.4 | 7 | — | — | |
| Nanobatching + PerseusParallelism=CP2TP4, µBatch Size=8, Sequence Length=8K2026.01 | 1.5 | 7.3 | — | — | |
| Megatron-LM + PerseusParallelism=CP2TP4, µBatch Size=8, Sequence Length=4K2026.01 | 0.2 | 7.4 | — | — | |
| Nanobatching + PerseusParallelism=CP2TP4, µBatch Size=8, Sequence Length=4K2026.01 | 0 | 7.3 | — | — | |
| Megatron-LM + PerseusParallelism=TP8, µBatch Size=8, Sequence Length=4K2026.01 | -0.3 | 10 | — | — | |
| Megatron-LM + PerseusParallelism=CP2TP4, µBatch Size=16, Sequence Length=4K2026.01 | -0.7 | 8.9 | — | — | |
| Megatron-LM + PerseusParallelism=CP2TP4, µBatch Size=8, Sequence Length=8K2026.01 | -1.4 | 11.5 | — | — | |
| KareusParallelism Strategy=TP8, Number of Microbatches=8, Sequence Length=4K2026.01 | — | — | 24.3 | 24 | |
| KareusParallelism Strategy=CP2TP4, Number of Microbatches=8, Sequence Length=4K2026.01 | — | — | 17.4 | 13.9 | |
| KareusParallelism Strategy=CP2TP4, Number of Microbatches=8, Sequence Length=8K2026.01 | — | — | 12.4 | 16.4 | |
| KareusParallelism Strategy=CP2TP4, Number of Microbatches=16, Sequence Length=4K2026.01 | — | — | 20.4 | 12.5 | |
| Nanobatching + PerseusParallelism Strategy=TP8, Number of Microbatches=8, Sequence Length=4K2026.01 | — | — | 21 | 18.9 | |
| Nanobatching + PerseusParallelism Strategy=CP2TP4, Number of Microbatches=8, Sequence Length=4K2026.01 | — | — | — | 7 | |
| Nanobatching + PerseusParallelism Strategy=CP2TP4, Number of Microbatches=8, Sequence Length=8K2026.01 | — | — | 4 | 4.4 | |
| Nanobatching + PerseusParallelism Strategy=CP2TP4, Number of Microbatches=16, Sequence Length=4K2026.01 | — | — | 6.6 | 5.1 |