Constrained LLM Decoding on Llama-2-70B
27.2Latency (ms)Pre³
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Pre³Batch Size=16, Hardware=4xH8002025.06 | 27.2 | 5.39 | |
| XGrammarBatch Size=16, Hardware=4xH8002025.06 | 28.75 | — | |
| Pre³Batch Size=64, Hardware=4xH8002025.06 | 48.48 | 14.85 | |
| Pre³Batch Size=32, Hardware=4xH8002025.06 | 54.24 | 1.6 | |
| XGrammarBatch Size=32, Hardware=4xH8002025.06 | 55.12 | — | |
| Pre³Batch Size=128, Hardware=4xH8002025.06 | 55.39 | 19.48 | |
| XGrammarBatch Size=64, Hardware=4xH8002025.06 | 56.94 | — | |
| XGrammarBatch Size=128, Hardware=4xH8002025.06 | 68.79 | — | |
| Pre³Batch Size=256, Hardware=4xH8002025.06 | 75.72 | 11.87 | |
| XGrammarBatch Size=256, Hardware=4xH8002025.06 | 85.92 | — |