Latency (ms) on Attention layer forward pass on Synthetic Attention Layer Input
145.1Latency (ms)Query Split (Flash-TPSP)
Evaluation Results
| Method | Links | |
|---|---|---|
| Query Split (Flash-TPSP)Data (B x S)=4 × 8192, Embedding Dimension=40962026.04 | 145.1 | |
| Query Split (Flash-TPSP)Data (B x S)=64 × 1024, Embedding Dimension=40962026.04 | 164.2 | |
| Row Parallel (Flash-TPSP)Data (B x S)=64 × 1024, Embedding Dimension=40962026.04 | 170.7 | |
| Row Parallel (Flash-TPSP)Data (B x S)=4 × 8192, Embedding Dimension=40962026.04 | 174.6 | |
| SlicingData (B x S)=64 × 1024, Embedding Dimension=40962026.04 | 179.6 | |
| Query Split (Flash-TPSP)Data (B x S)=32 × 2048, Embedding Dimension=40962026.04 | 181.2 | |
| BaselineData (B x S)=4 × 8192, Embedding Dimension=40962026.04 | 195.6 | |
| BaselineData (B x S)=64 × 1024, Embedding Dimension=40962026.04 | 196.2 | |
| Row Parallel (Flash-TPSP)Data (B x S)=32 × 2048, Embedding Dimension=40962026.04 | 200.7 | |
| SlicingData (B x S)=32 × 2048, Embedding Dimension=40962026.04 | 202.2 | |
| Query Split (Flash-TPSP)Data (B x S)=16 × 4096, Embedding Dimension=40962026.04 | 215.8 | |
| BaselineData (B x S)=32 × 2048, Embedding Dimension=40962026.04 | 222.8 | |
| Row Parallel (Flash-TPSP)Data (B x S)=16 × 4096, Embedding Dimension=40962026.04 | 230.9 | |
| SlicingData (B x S)=16 × 4096, Embedding Dimension=40962026.04 | 246.1 | |
| BaselineData (B x S)=16 × 4096, Embedding Dimension=40962026.04 | 270.1 | |
| SlicingData (B x S)=4 × 8192, Embedding Dimension=40962026.04 | 279.7 |