Serving Latency on MATH500, GSM8K, SharedGPT, and HumanEval (test)
9.3Overhead (%)MarginGate
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MarginGateModel=Llama-3.2-1B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | 9.3 | — | |
| MarginGateModel=Qwen2.5-14B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | 23.9 | — | |
| MarginGateModel=Llama-3.1-8B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | 29 | — | |
| LLM-42Model=Qwen2.5-14B, Hardware=A6000, Batch Size=8, Output Token Cap=1024, Verifier setting K=642026.05 | 47.5 | — | |
| MarginGateModel=Qwen2.5-7B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | 49.8 | — | |
| LLM-42Model=Qwen2.5-7B, Hardware=A6000, Batch Size=8, Output Token Cap=1024, Verifier setting K=642026.05 | 50.7 | — | |
| LLM-42Model=Llama-3.1-8B, Hardware=A6000, Batch Size=8, Output Token Cap=1024, Verifier setting K=642026.05 | 64.6 | — | |
| LLM-42Model=Llama-3.2-1B, Hardware=A6000, Batch Size=8, Output Token Cap=1024, Verifier setting K=642026.05 | 69.9 | — | |
| LLM-42Model=DSR1-Dist.-Qwen-7B, Hardware=A6000, Batch Size=8, Output Token Cap=1024, Verifier setting K=642026.05 | 72 | — | |
| MarginGateModel=DSR1-Dist.-Qwen-7B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | 84.5 | — | |
| Batch-InvariantModel=Llama-3.2-1B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | 119 | — | |
| Batch-InvariantModel=Qwen2.5-7B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | 129 | — | |
| Batch-InvariantModel=Qwen2.5-14B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | 129 | — | |
| Batch-InvariantModel=DSR1-Dist.-Qwen-7B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | 133 | — | |
| LayerCastModel=Llama-3.2-1B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | 133.4 | — | |
| Batch-InvariantModel=Llama-3.1-8B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | 143 | — | |
| LayerCastModel=DSR1-Dist.-Qwen-7B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | 345.1 | — | |
| LayerCastModel=Qwen2.5-7B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | 346.9 | — | |
| LayerCastModel=Llama-3.1-8B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | 354.3 | — | |
| LayerCastModel=Qwen2.5-14B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | 373.6 | — | |
| BF16Model=Llama-3.2-1B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | — | 2.22 | |
| BF16Model=Llama-3.1-8B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | — | 12.27 | |
| BF16Model=Qwen2.5-7B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | — | 12.79 | |
| BF16Model=Qwen2.5-14B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | — | 22.67 | |
| BF16Model=DSR1-Dist.-Qwen-7B, Hardware=A6000, Batch Size=8, Output Token Cap=10242026.05 | — | 23.06 |