Inference Throughput on LLaMA-3 8B
1,020Decode Throughput (tok/s)IQ3_S
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| IQ3_SHardware=RTX 5090, Base Model=LLaMA-3 8B, Batch size (decode)=1, Batch size (prefill)=322026.03 | 1,020 | 47,800 | 2.1 | |
| ITQ3_SHardware=RTX 5090, Base Model=LLaMA-3 8B, Batch size (decode)=1, Batch size (prefill)=322026.03 | 960 | 51,200 | 2 | |
| Q4_K_MHardware=RTX 5090, Base Model=LLaMA-3 8B, Batch size (decode)=1, Batch size (prefill)=322026.03 | 890 | 42,100 | 1.9 | |
| FP16Hardware=RTX 5090, Base Model=LLaMA-3 8B, Batch size (decode)=1, Batch size (prefill)=322026.03 | 480 | 28,400 | 1 |