LLM Decoding on LLaMA 128K context 3.1-8B
72.8Dense Latency (ms)Dense
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| DenseModel=LLaMA 3.1-8B, Hardware=RTX PRO 6000 WS, Scratch cache=2048-block, Timed decode steps=5002026.05 | 72.8 | — | — | — | — | — | |
| Runtime-Certified Bounded-Error Quantized AttentionModel=LLaMA 3.1-8B, Hardware=RTX PRO 6000 WS, Scratch cache=2048-block, Timed decode steps=5002026.05 | — | 346.7 | 4.76 | 226 | 87.7 | 644 |