List your company

ResearchFormerly WizWand

Decoding Latency

Benchmarks

DatasetSOTA methodMetricTrendResultsLast updated

Synthetic Context Sequences (test)

0.019Latency (16k Context)

16

Feb 26, 2026

Llama-3.1-8B 16k sequence length v1 (inference)

0.024Decoding latency (s)

8

Feb 26, 2026

Llama-3.1-8B 32k sequence length v1 (inference)

0.033Decoding Latency (s)

7

Feb 26, 2026

Llama-2-7B 32k sequence length v1 (inference)

0.062Decoding Latency (s)

6

Feb 26, 2026

Llama-2-7B 16k sequence length v1 (inference)

0.041Decoding Latency (s)

6

Feb 26, 2026

Llama-3.1-8B 64k sequence length v1

0.05Decoding Latency (s)

5

Feb 26, 2026

Llama-2-7B 64k sequence length v1 (inference)

0.098Decoding Latency (s)

5

Feb 26, 2026

Llama-2-7B 96k sequence length v1 (inference)

0.115Decoding Latency (s)

4

Feb 26, 2026