Constrained LLM Decoding on DeepSeek-V2-Lite-Chat 15.7B
49.91Inference Time (ms)Pre³
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Pre³Batch Size=16, Hardware=2xH8002025.06 | 49.91 | 3.57 | |
| XGrammarBatch Size=16, Hardware=2xH8002025.06 | 51.76 | — | |
| Pre³Batch Size=32, Hardware=2xH8002025.06 | 53.71 | 9.65 | |
| Pre³Batch Size=64, Hardware=2xH8002025.06 | 54.41 | 30.01 | |
| XGrammarBatch Size=32, Hardware=2xH8002025.06 | 59.45 | — | |
| Pre³Batch Size=128, Hardware=2xH8002025.06 | 61.63 | 40.78 | |
| Pre³Batch Size=256, Hardware=2xH8002025.06 | 75.47 | 37.86 | |
| XGrammarBatch Size=64, Hardware=2xH8002025.06 | 77.74 | — | |
| XGrammarBatch Size=128, Hardware=2xH8002025.06 | 104.06 | — | |
| XGrammarBatch Size=256, Hardware=2xH8002025.06 | 121.46 | — |