System Performance Evaluation on Multi-LLM Serving Workload
96.4TTFT (ms)Full-FT
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Full-FTModel=Qwen3-8B-Base, # Bits=16/16, Decode Execution=Independent2026.03 | 96.4 | 12.2 | 81.3 | |
| QSUNModel=Qwen3-8B-Base, # Bits=16/4, Decode Execution=Shared2026.03 | 97.1 | 6.6 | 147.9 | |
| QSUNModel=LLaMA3.1-8B, # Bits=16/4, Decode Execution=Shared2026.03 | 98.9 | 7.6 | 130.2 | |
| Full-FTModel=LLaMA3.1-8B, # Bits=16/16, Decode Execution=Independent2026.03 | 99.1 | 13.8 | 73.3 | |
| AWQModel=Qwen3-8B-Base, # Bits=4/4, Decode Execution=Independent2026.03 | 115.8 | 6.6 | 148.9 | |
| AWQModel=LLaMA3.1-8B, # Bits=4/4, Decode Execution=Independent2026.03 | 118.7 | 7.6 | 130.8 |