Training Efficiency on Yuan3.0-1T Pre-training Base (train)
92.6TFLOPSLAEP
Evaluation results
| Method | Links | |
|---|---|---|
| LAEPTotal Params=1010B, Layers=103, Experts=48*2026.01 | 92.6 | |
| DeepSeek-V3 Sequence-Wise Auxiliary LossTotal Params=1142B, Layers=103, Experts=482026.01 | 84.74 | |
| LAEP w/o rearrangementTotal Params=1010B, Layers=103, Experts=48*2026.01 | 82.25 | |
| DeepSeek-V3 Sequence-Wise Auxiliary LossTotal Params=1515B, Layers=103, Experts=642026.01 | 80.82 | |
| Mixtral Auxiliary Load Balancing LossTotal Params=1515B, Layers=103, Experts=642026.01 | 80.36 | |
| Base modelTotal Params=1515B, Layers=103, Experts=642026.01 | 62.14 |