Constrained LLM Decoding on Llama-3-8B
11.77Inference Time (ms)Pre³
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Pre³Batch Size=16, Hardware=2xH8002025.06 | 11.77 | 22.49 | |
| XGrammarBatch Size=16, Hardware=2xH8002025.06 | 15.19 | — | |
| Pre³Batch Size=32, Hardware=2xH8002025.06 | 31.12 | 28.78 | |
| Pre³Batch Size=64, Hardware=2xH8002025.06 | 35.88 | 31.09 | |
| XGrammarBatch Size=32, Hardware=2xH8002025.06 | 43.69 | — | |
| Pre³Batch Size=128, Hardware=2xH8002025.06 | 45.32 | 30.5 | |
| XGrammarBatch Size=64, Hardware=2xH8002025.06 | 52.07 | — | |
| Pre³Batch Size=256, Hardware=2xH8002025.06 | 64.42 | 29.2 | |
| XGrammarBatch Size=128, Hardware=2xH8002025.06 | 65.21 | — | |
| XGrammarBatch Size=256, Hardware=2xH8002025.06 | 90.98 | — | |
| Pre³Batch Size=512, Hardware=2xH8002025.06 | 104.46 | 29.24 | |
| XGrammarBatch Size=512, Hardware=2xH8002025.06 | 147.64 | — | |
| Pre³Batch Size=1024, Hardware=2xH8002025.06 | 201.16 | 26.25 | |
| XGrammarBatch Size=1024, Hardware=2xH8002025.06 | 272.77 | — |