Multi-step LLM Interaction on DeepSeek-R1-Distill-Llama-8B Inference Performance (test)
100High-Precision RatioBF16
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| BF16Precision=BF162026.02 | 100 | 62.7 | 18.7 | |
| Random@80%Sampling Ratio=80%2026.02 | 80 | 54.2 | 12.75 | |
| Random@60%Sampling Ratio=60%2026.02 | 60 | 51.6 | 12.67 | |
| Random@40%Sampling Ratio=40%2026.02 | 40 | 48.1 | 10.25 | |
| Random@20%Sampling Ratio=20%2026.02 | 20 | 45.3 | 6.5 | |
| Router(Ours)Routing=Dynamic Mix Precision2026.02 | 9.8 | 47.8 | 38.77 | |
| GPTQ-Int4Quantization=4-bit2026.02 | 0 | 44 | — |