LLM Training Performance on Qwen models on RTX 3090 (24GB VRAM) 2.5 (train)
35.09TFLOPSMegaTrain
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| MegaTrainModel=Qwen2.5-7B, Max BS=52026.04 | 35.09 | 22.63 | 56.7 | 768 | |
| MegaTrainModel=Qwen2.5-3B, Max BS=72026.04 | 33.18 | 22.83 | 25 | 1,792 | |
| MegaTrainModel=Qwen2.5-14B, Max BS=32026.04 | 30.19 | 21.1 | 103.7 | 341 | |
| ZeRO-3 OffloadModel=Qwen2.5-7B, Max BS=12026.04 | 27.49 | 20.83 | — | — | |
| ZeRO-3 OffloadModel=Qwen2.5-3B, Max BS=12026.04 | 23.91 | 20.32 | — | — |