Inference Throughput on Llama Instruct 3.1-8B (internal harness)
6,991Throughput (TPS)Pre-compressed only
Evaluation Results
| Method | Links | |
|---|---|---|
| Pre-compressed onlyMethod=Pre-compression2026.03 | 6,991 | |
| Pre-comp. + clusteringMethod=Pre-comp. + K-means (rebuild-to-dense)2026.03 | 6,872 | |
| AWQ INT4Method=Uniform quant.2026.03 | 4,606 | |
| FP8Method=Uniform quant.2026.03 | 3,316 | |
| Baseline (FP16)Method=Full precision2026.03 | 2,194 | |
| Clustering (Orig)Method=K-means (rebuild-to-dense)2026.03 | 2,194 | |
| GPTQ INT4Method=Uniform quant.2026.03 | 1,812 | |
| AQLM 2-bitMethod=Codebook2026.03 | 1,023 |