Question Answering on RAG QA 5 retrieved documents
191.7TTFT (ms)Lazy-Attn
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Lazy-AttnModel=Tulu3-Block-FT 8B2026.06 | 191.7 | 6.2 | |
| Lazy-AttnModel=Llama-3.1-70B, Tensor Parallelism (TP)=42026.06 | 238.4 | 5.2 | |
| CacheBlendModel=Tulu3-Block-FT 8B2026.06 | 274.8 | 4.4 | |
| CacheBlendModel=Llama-3.1-70B, Tensor Parallelism (TP)=42026.06 | 365.5 | 3.4 | |
| Standard RAGModel=Tulu3-Block-FT 8B2026.06 | 1,196.8 | 1 | |
| Standard RAGModel=Llama-3.1-70B, Tensor Parallelism (TP)=42026.06 | 1,253 | 1 |