Prefilling Profiling on WikiText (test)
38Time (s)W4A4
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| W4A4Model=Llama-3-8B, bank_size=642025.12 | 38 | 40.61 | 1.26 | |
| SQ-formatModel=Llama-3-8B, Sparsity=0.875, bank_size=642025.12 | 41 | 38.19 | 1.17 | |
| SQ-formatModel=Llama-3-8B, Sparsity=0.75, bank_size=642025.12 | 42 | 36.9 | 1.14 | |
| SQ-formatModel=Llama-3-8B, Sparsity=0.5, bank_size=642025.12 | 45 | 35.75 | 1.07 | |
| W4A8Model=Llama-3-8B, bank_size=642025.12 | 48 | 34.69 | 1 | |
| W4A8Model=Llama-3-70B, bank_size=642025.12 | 108 | 10.84 | 1 | |
| W4A4Model=Llama-3-70B, bank_size=642025.12 | 516 | 20.86 | 1.92 | |
| SQ-formatModel=Llama-3-70B, Sparsity=0.875, bank_size=642025.12 | 555 | 18.54 | 1.71 | |
| SQ-formatModel=Llama-3-70B, Sparsity=0.75, bank_size=642025.12 | 629 | 16.94 | 1.56 | |
| SQ-formatModel=Llama-3-70B, Sparsity=0.5, bank_size=642025.12 | 742 | 14.27 | 1.32 |