Spec-Bench
Benchmarks
TaskDatasetSOTA resultTrendResults
Spec-Bench
195.6MT Score
57
Spec-Bench
5.31Speedup
53
Spec-Bench
2.05MT Score
21
Spec-Bench Overall
2.33SD Score
21
Spec-Bench Trans.
6.41CR
21
Spec-Bench Sum.
4.73CR Score
21
Spec-Bench RAG
5.48CR
21
Spec-Bench QA
4.54CR
21
Spec-Bench Multi.
3.22CR
21
Spec-Bench Math
4.09CR
21
Spec-Bench
267.6Conv. Acc
11
Spec-Bench
519.7Throughput (Conv.)
10
Spec-Bench OLMo 2 7B
5.12Conversation Score
5
Spec-Bench Llama2-7B v1.0 (test)
2.73MT Score
4
Spec-Bench (test)
—Throughput (Conv.)
0