Constrained LLM Decoding on Qwen2-14B INT8
16.52Inference Time (ms)Pre³
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Pre³Batch Size=16, Hardware=2xH800, Precision=INT82025.06 | 16.52 | 1.52 | |
| XGrammarBatch Size=16, Hardware=2xH800, Precision=INT82025.06 | 16.77 | — | |
| Pre³Batch Size=32, Hardware=2xH800, Precision=INT82025.06 | 43.09 | 10.12 | |
| XGrammarBatch Size=32, Hardware=2xH800, Precision=INT82025.06 | 47.94 | — | |
| Pre³Batch Size=64, Hardware=2xH800, Precision=INT82025.06 | 49.99 | 12.37 | |
| XGrammarBatch Size=64, Hardware=2xH800, Precision=INT82025.06 | 57.05 | — | |
| Pre³Batch Size=128, Hardware=2xH800, Precision=INT82025.06 | 65.5 | 12.14 | |
| XGrammarBatch Size=128, Hardware=2xH800, Precision=INT82025.06 | 74.54 | — | |
| Pre³Batch Size=256, Hardware=2xH800, Precision=INT82025.06 | 80.34 | 18.55 | |
| XGrammarBatch Size=256, Hardware=2xH800, Precision=INT82025.06 | 98.64 | — | |
| Pre³Batch Size=512, Hardware=2xH800, Precision=INT82025.06 | 143.83 | 11.47 | |
| XGrammarBatch Size=512, Hardware=2xH800, Precision=INT82025.06 | 162.47 | — | |
| Pre³Batch Size=1024, Hardware=2xH800, Precision=INT82025.06 | 232.18 | 18.65 | |
| XGrammarBatch Size=1024, Hardware=2xH800, Precision=INT82025.06 | 285.42 | — |