LLM Training Optimization on Qwen 3 1.7B
0.149Time ReductionKareus
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| KareusParallelism=TP8, µBatch Size=8, Sequence Length=8K2026.01 | 0.149 | 0.151 | |
| KareusParallelism=TP8, µBatch Size=16, Sequence Length=4K2026.01 | 0.148 | 0.173 | |
| KareusParallelism=TP8, µBatch Size=8, Sequence Length=4K2026.01 | 0.122 | 0.221 | |
| KareusParallelism=CP2TP4, µBatch Size=16, Sequence Length=4K2026.01 | 0.105 | 0.112 | |
| Nanobatching + PerseusParallelism=TP8, µBatch Size=16, Sequence Length=4K2026.01 | 0.094 | 0.134 | |
| Nanobatching + PerseusParallelism=TP8, µBatch Size=8, Sequence Length=8K2026.01 | 0.09 | 0.122 | |
| KareusParallelism=CP2TP4, µBatch Size=8, Sequence Length=8K2026.01 | 0.08 | 0.087 | |
| Nanobatching + PerseusParallelism=TP8, µBatch Size=8, Sequence Length=4K2026.01 | 0.056 | 0.128 | |
| Nanobatching + PerseusParallelism=CP2TP4, µBatch Size=16, Sequence Length=4K2026.01 | 0.039 | 0.069 | |
| Nanobatching + PerseusParallelism=CP2TP4, µBatch Size=8, Sequence Length=8K2026.01 | 0.017 | 0.067 | |
| Megatron-LM + PerseusParallelism=CP2TP4, µBatch Size=8, Sequence Length=8K2026.01 | 0.005 | 0.074 | |
| Megatron-LM + PerseusParallelism=TP8, µBatch Size=8, Sequence Length=8K2026.01 | 0.001 | 0.079 | |
| Megatron-LM + PerseusParallelism=CP2TP4, µBatch Size=16, Sequence Length=4K2026.01 | 0 | 0.069 | |
| Megatron-LM + PerseusParallelism=TP8, µBatch Size=16, Sequence Length=4K2026.01 | -0.001 | 0.076 | |
| Megatron-LM + PerseusParallelism=TP8, µBatch Size=8, Sequence Length=4K2026.01 | -0.005 | 0.077 | |
| Megatron-LM + PerseusParallelism=CP2TP4, µBatch Size=8, Sequence Length=4K2026.01 | -0.005 | 0.071 | |
| KareusParallelism=CP2TP4, µBatch Size=8, Sequence Length=4K2026.01 | -0.005 | 0.071 | |
| Nanobatching + PerseusParallelism=CP2TP4, µBatch Size=8, Sequence Length=4K2026.01 | -0.204 | 0.031 |