LLM Decoding on LLaMA 32K context 3.1-8B
61.2Dense Latency (ms)Dense
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| DenseModel=LLaMA 3.1-8B, Hardware=RTX PRO 6000 WS, Scratch cache=2048-block, Timed decode steps=5002026.05 | 61.2 | — | — | — | — | — | |
| Runtime-Certified Bounded-Error Quantized AttentionModel=LLaMA 3.1-8B, Hardware=RTX PRO 6000 WS, Scratch cache=2048-block, Timed decode steps=5002026.05 | — | 232.2 | 3.79 | 217 | 99.7 | 114 |