Inference Efficiency on 90k Context Length Llama-3.1-8B
8.9Throughput (queries/s)Finetuning
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| FinetuningModel=Llama-3.1-8B, GPU=single A1002025.03 | 8.9 | — | — | — | |
| DBSAModel=Llama-3.1-8B, GPU=single A1002025.03 | 7.7 | — | — | — | |
| Fixed ICLCaching Strategy=cached, Model=Llama-3.1-8B, GPU=single A1002025.03 | 6.8 | — | — | — | |
| RetICLCaching Strategy=no cache, Model=Llama-3.1-8B, GPU=single A1002025.03 | 0.4 | — | — | — | |
| DBSAModel=Llama-3.1-8B, Hardware=Single A100 GPU, Context Length=90k2025.03 | — | — | 4 | 0.053 | |
| FinetuningModel=Llama-3.1-8B, Hardware=Single A100 GPU, Fine-tuning Protocol=LoRA, Context Length=90k2025.03 | — | — | — | 0.046 | |
| Fixed ICLModel=Llama-3.1-8B, Hardware=Single A100 GPU, Caching Strategy=cached, Context Length=90k2025.03 | — | — | 6.5 | 0.06 | |
| RetICLModel=Llama-3.1-8B, Hardware=Single A100 GPU, Caching Strategy=no cache, Context Length=90k2025.03 | — | — | 1 | 1 |