Decode Throughput on BenchRandom
269.1Decode Throughput (tok/s)NVE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| NVEModel=Llama-3.2-1B, Quantization format=W4A8, Hardware=T4 GPU, Batch size=1, Iterations=2,0002026.04 | 269.1 | — | |
| llama.cppModel=Llama-3.2-1B, Quantization format=Q4_0, Hardware=T4 GPU, Batch size=1, Iterations=2,0002026.04 | 150.8 | — | |
| NVEModel=Llama-3.2-1B, Quantization format=W4A16, Hardware=T4 GPU, Batch size=1, Iterations=2,0002026.04 | 116.7 | — | |
| NVEModel=Llama-3.2-3B, Quantization format=W4A8, Hardware=T4 GPU, Batch size=1, Iterations=2,0002026.04 | 108.8 | — | |
| llama.cppModel=Llama-3.2-3B, Quantization format=Q4_0, Hardware=T4 GPU, Batch size=1, Iterations=2,0002026.04 | 70.9 | — | |
| NVEModel=Llama-3.1-8B, Quantization format=W4A8, Hardware=T4 GPU, Batch size=1, Iterations=2,0002026.04 | 47.7 | — | |
| NVEModel=Llama-3.2-3B, Quantization format=W4A16, Hardware=T4 GPU, Batch size=1, Iterations=2,0002026.04 | 43 | — | |
| llama.cppModel=Llama-3.1-8B, Quantization format=Q4_0, Hardware=T4 GPU, Batch size=1, Iterations=2,0002026.04 | 30.8 | — | |
| NVEModel=Llama-3.1-8B, Quantization format=W4A16, Hardware=T4 GPU, Batch size=1, Iterations=2,0002026.04 | 16.7 | — |