LLM Decoding on LLaMA 64K context 3.1-8B
62.6Dense Latency (ms)Dense
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| DenseModel=LLaMA 3.1-8B, Hardware=RTX PRO 6000 WS, Scratch cache=2048-block, Timed decode steps=5002026.05 | 62.6 | — | — | — | — | — | |
| Runtime-Certified Bounded-Error Quantized AttentionModel=LLaMA 3.1-8B, Hardware=RTX PRO 6000 WS, Scratch cache=2048-block, Timed decode steps=5002026.05 | — | 257.4 | 4.11 | 222 | 99.6 | 251 |