MLP Layer Forward Pass Performance on MLP Layer d_model=4096 Input shape=(32, 4096, 4096)
76.9Forward Compute Time (ms)Data Slicing Approach
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Data Slicing Approachd_model=4096, Tensor Parallelism (TP)=4, Input tensor shape (b, s, d)=(32, 4096, 4096), Averaged iterations=102026.04 | 76.9 | 7.1 | 83.9 | 83.9 | 31.4 | |
| Flash-RSd_model=4096, Tensor Parallelism (TP)=4, Input tensor shape (b, s, d)=(32, 4096, 4096), Averaged iterations=10, primitive=Flash-RS, framework=Flash-Overlap2026.04 | 77 | 0.1 | 77.1 | 99.8 | 36.9 | |
| Vanilla Baselined_model=4096, Tensor Parallelism (TP)=4, Input tensor shape (b, s, d)=(32, 4096, 4096), Averaged iterations=102026.04 | 78.5 | 43.8 | 122.3 | — | — |