Fused CUDA kernel latency evaluation on normalized synthetic vectors
6.1Latency (µs)IsoQuant-Fast
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| IsoQuant-Fastdtype=fp16, bits=3, dim=128, batch size=81922026.03 | 6.1 | 6 | |
| IsoQuant-Fulldtype=fp16, bits=3, dim=128, batch size=81922026.03 | 6.2 | 5.92 | |
| IsoQuant-2Ddtype=fp16, bits=3, dim=128, batch size=81922026.03 | 6.2 | 5.92 | |
| IsoQuant-Fulldtype=fp32, bits=3, dim=128, batch size=81922026.03 | 6.9 | 5.07 | |
| IsoQuant-2Ddtype=fp32, bits=3, dim=256, batch size=81922026.03 | 6.9 | 5.47 | |
| IsoQuant-2Ddtype=fp32, bits=2, dim=128, batch size=81922026.03 | 7 | 4.77 | |
| IsoQuant-Fastdtype=fp32, bits=3, dim=256, batch size=81922026.03 | 7 | 5.45 | |
| IsoQuant-Fastdtype=fp32, bits=2, dim=128, batch size=81922026.03 | 7.1 | 4.68 | |
| IsoQuant-2Ddtype=fp32, bits=3, dim=128, batch size=81922026.03 | 7.1 | 4.91 | |
| IsoQuant-Fastdtype=fp32, bits=2, dim=256, batch size=81922026.03 | 7.2 | 4.66 | |
| IsoQuant-2Ddtype=fp32, bits=2, dim=256, batch size=81922026.03 | 7.2 | 4.66 | |
| IsoQuant-Fulldtype=fp32, bits=3, dim=256, batch size=81922026.03 | 7.2 | 5.29 | |
| IsoQuant-Fulldtype=fp32, bits=2, dim=128, batch size=81922026.03 | 7.3 | 4.6 | |
| IsoQuant-Fulldtype=fp32, bits=2, dim=256, batch size=81922026.03 | 7.4 | 4.59 | |
| IsoQuant-Fastdtype=fp16, bits=4, dim=256, batch size=81922026.03 | 7.5 | 6.24 | |
| IsoQuant-2Ddtype=fp16, bits=4, dim=256, batch size=81922026.03 | 7.5 | 6.2 | |
| IsoQuant-2Ddtype=fp32, bits=4, dim=256, batch size=81922026.03 | 7.5 | 6.39 | |
| IsoQuant-Fastdtype=fp32, bits=3, dim=128, batch size=81922026.03 | 7.6 | 4.59 | |
| IsoQuant-Fastdtype=fp32, bits=4, dim=256, batch size=81922026.03 | 7.6 | 6.31 | |
| IsoQuant-Fulldtype=fp16, bits=4, dim=256, batch size=81922026.03 | 8.1 | 5.76 | |
| IsoQuant-Fulldtype=fp32, bits=4, dim=256, batch size=81922026.03 | 8.1 | 5.92 | |
| IsoQuant-Fastdtype=fp16, bits=2, dim=128, batch size=81922026.03 | 8.2 | 3.98 | |
| IsoQuant-Fulldtype=fp32, bits=4, dim=128, batch size=81922026.03 | 8.2 | 5.49 | |
| IsoQuant-Fastdtype=fp32, bits=4, dim=128, batch size=81922026.03 | 8.2 | 5.48 | |
| IsoQuant-Fastdtype=fp16, bits=2, dim=256, batch size=81922026.03 | 8.3 | 3.92 | |
| IsoQuant-2Ddtype=fp16, bits=2, dim=512, batch size=81922026.03 | 8.3 | 4.31 | |
| IsoQuant-2Ddtype=fp32, bits=4, dim=128, batch size=81922026.03 | 8.3 | 5.4 | |
| IsoQuant-2Ddtype=fp16, bits=2, dim=256, batch size=81922026.03 | 8.4 | 3.89 | |
| IsoQuant-Fulldtype=fp16, bits=2, dim=512, batch size=81922026.03 | 8.4 | 4.27 | |
| IsoQuant-Fastdtype=fp16, bits=3, dim=512, batch size=81922026.03 | 8.4 | 4.73 | |
| IsoQuant-Fulldtype=fp16, bits=2, dim=128, batch size=81922026.03 | 8.5 | 3.86 | |
| IsoQuant-2Ddtype=fp16, bits=2, dim=128, batch size=81922026.03 | 8.5 | 3.85 | |
| IsoQuant-Fastdtype=fp16, bits=2, dim=512, batch size=81922026.03 | 8.5 | 4.23 | |
| IsoQuant-Fulldtype=fp16, bits=2, dim=256, batch size=81922026.03 | 8.8 | 3.72 | |
| IsoQuant-2Ddtype=fp16, bits=3, dim=256, batch size=81922026.03 | 9.2 | 3.99 | |
| IsoQuant-Fastdtype=fp16, bits=4, dim=128, batch size=81922026.03 | 9.4 | 4.71 | |
| IsoQuant-2Ddtype=fp16, bits=4, dim=128, batch size=81922026.03 | 9.4 | 4.7 | |
| IsoQuant-Fastdtype=fp16, bits=3, dim=256, batch size=81922026.03 | 9.5 | 3.88 | |
| IsoQuant-Fulldtype=fp16, bits=4, dim=128, batch size=81922026.03 | 9.6 | 4.62 | |
| IsoQuant-2Ddtype=fp16, bits=3, dim=512, batch size=81922026.03 | 9.6 | 4.15 | |
| IsoQuant-Fulldtype=fp16, bits=3, dim=256, batch size=81922026.03 | 9.7 | 3.8 | |
| IsoQuant-Fulldtype=fp16, bits=3, dim=512, batch size=81922026.03 | 9.9 | 4.05 | |
| IsoQuant-2Ddtype=fp32, bits=2, dim=512, batch size=81922026.03 | 10.5 | 3.6 | |
| IsoQuant-2Ddtype=fp32, bits=3, dim=512, batch size=81922026.03 | 11 | 4.02 | |
| IsoQuant-Fastdtype=fp32, bits=2, dim=512, batch size=81922026.03 | 11.4 | 3.32 | |
| IsoQuant-Fastdtype=fp32, bits=3, dim=512, batch size=81922026.03 | 11.5 | 3.85 | |
| IsoQuant-Fulldtype=fp32, bits=2, dim=512, batch size=81922026.03 | 12.1 | 3.11 | |
| IsoQuant-Fastdtype=fp16, bits=4, dim=512, batch size=81922026.03 | 12.5 | 4.02 | |
| IsoQuant-Fulldtype=fp32, bits=3, dim=512, batch size=81922026.03 | 12.5 | 3.54 | |
| IsoQuant-2Ddtype=fp16, bits=4, dim=512, batch size=81922026.03 | 13.4 | 3.77 | |
| IsoQuant-Fastdtype=fp32, bits=4, dim=512, batch size=81922026.03 | 13.5 | 3.91 | |
| IsoQuant-Fulldtype=fp16, bits=4, dim=512, batch size=81922026.03 | 13.9 | 3.63 | |
| IsoQuant-2Ddtype=fp32, bits=4, dim=512, batch size=81922026.03 | 13.9 | 3.79 | |
| IsoQuant-Fulldtype=fp32, bits=4, dim=512, batch size=81922026.03 | 14.8 | 3.56 | |
| RotorQuantdtype=fp16, bits=2, dim=256, batch size=81922026.03 | 32.6 | — | |
| RotorQuantdtype=fp16, bits=2, dim=128, batch size=81922026.03 | 32.7 | — | |
| RotorQuantdtype=fp32, bits=2, dim=128, batch size=81922026.03 | 33.4 | — | |
| RotorQuantdtype=fp32, bits=2, dim=256, batch size=81922026.03 | 33.8 | — | |
| RotorQuantdtype=fp32, bits=3, dim=128, batch size=81922026.03 | 35.1 | — | |
| RotorQuantdtype=fp16, bits=2, dim=512, batch size=81922026.03 | 36 | — | |
| RotorQuantdtype=fp16, bits=3, dim=128, batch size=81922026.03 | 36.4 | — | |
| RotorQuantdtype=fp16, bits=3, dim=256, batch size=81922026.03 | 36.8 | — | |
| RotorQuantdtype=fp32, bits=2, dim=512, batch size=81922026.03 | 37.8 | — | |
| RotorQuantdtype=fp32, bits=3, dim=256, batch size=81922026.03 | 37.9 | — | |
| RotorQuantdtype=fp16, bits=3, dim=512, batch size=81922026.03 | 39.9 | — | |
| RotorQuantdtype=fp32, bits=3, dim=512, batch size=81922026.03 | 44.1 | — | |
| RotorQuantdtype=fp16, bits=4, dim=128, batch size=81922026.03 | 44.2 | — | |
| RotorQuantdtype=fp32, bits=4, dim=128, batch size=81922026.03 | 44.7 | — | |
| RotorQuantdtype=fp16, bits=4, dim=256, batch size=81922026.03 | 46.7 | — | |
| RotorQuantdtype=fp32, bits=4, dim=256, batch size=81922026.03 | 47.9 | — | |
| RotorQuantdtype=fp16, bits=4, dim=512, batch size=81922026.03 | 50.4 | — | |
| RotorQuantdtype=fp32, bits=4, dim=512, batch size=81922026.03 | 52.9 | — |