LLM Decoding on LLaMA 8K context 3.1-8B
60.9Dense Latency (ms)Dense
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| DenseModel=LLaMA 3.1-8B, Hardware=RTX PRO 6000 WS, Scratch cache=2048-block, Timed decode steps=5002026.05 | 60.9 | — | — | — | — | — | |
| Runtime-Certified Bounded-Error Quantized AttentionModel=LLaMA 3.1-8B, Hardware=RTX PRO 6000 WS, Scratch cache=2048-block, Timed decode steps=5002026.05 | — | 166.3 | 2.73 | 159 | 100 | 0 |