LLM Inference Efficiency on Instruction-in-Wild self-instructed prompts 36
2.52Throughput (samples/s)Ground Truth Preditor
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Ground Truth PreditorResponse length perception module=Ground Truth2023.05 | 2.52 | 107 | 201 | |
| Instruction Tunning (max)Response length perception module=Instruction Tunning (max)2023.05 | 2.27 | 86 | 208 | |
| [LEN]-token Fine-tuneResponse length perception module=[LEN]-token Fine-tune2023.05 | 2.1 | 72 | 210 | |
| Pooling + MLPResponse length perception module=Pooling + MLP2023.05 | 1.96 | 61 | 216 | |
| Instruction Tunning (mean)Response length perception module=Instruction Tunning (mean)2023.05 | 1.77 | 45 | 211 | |
| Perception Only*Response length perception module=Perception Only*2023.05 | 1.4 | 15 | 328 | |
| VanillaResponse length perception module=None (Baseline)2023.05 | 1.22 | — | 377 |