Tokenization Disambiguation on IMDB English
6.58Average Perplexity (PPL)Verification-based
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Verification-basedk=32, Backbone=Llama-3.1-8B2026.04 | 6.58 | 0 | 0 | 2.32 | 86 | 15.17 | 209.59 | 100 | |
| MWISk=32, Backbone=Llama-3.1-8B2026.04 | 6.87 | 0.09 | 0.75 | 2.12 | 81 | 7.64 | 55.92 | 100 | |
| Discopk=32, Backbone=Llama-3.1-8B2026.04 | 7.26 | 0 | 0 | 2.28 | 88 | 4.9 | 0 | — | |
| ReTokSynck=32, Backbone=Llama-3.1-8B2026.04 | 7.26 | 0 | 0 | 2.28 | 88 | 4.94 | 0.82 | 100 | |
| Stepwise Verificationk=32, Backbone=Llama-3.1-8B2026.04 | 7.36 | 0 | 0.05 | 2.31 | 89 | 6.65 | 35.71 | 100 | |
| Syncpoolk=32, Backbone=Llama-3.1-8B2026.04 | 7.65 | 0 | 0 | 1.68 | 64 | 5.05 | 3.06 | 100 | |
| MWISk=128, Backbone=Llama-3.1-8B2026.04 | 9.34 | 0.11 | 0.75 | 2.61 | 82 | 9.01 | 80.92 | 100 | |
| Verification-basedk=128, Backbone=Llama-3.1-8B2026.04 | 11 | 0 | 0 | 2.82 | 86 | 441.21 | 8,759.64 | 100 | |
| Discopk=128, Backbone=Llama-3.1-8B2026.04 | 11.33 | 0 | 0 | 2.95 | 91 | 4.98 | 0 | — | |
| ReTokSynck=128, Backbone=Llama-3.1-8B2026.04 | 11.36 | 0 | 0 | 2.95 | 91 | 5.05 | 1.41 | 99.9 | |
| MWISk=512, Backbone=Llama-3.1-8B2026.04 | 11.74 | 0.15 | 0.83 | 2.93 | 80 | 23.89 | 361.2 | 100 | |
| Stepwise Verificationk=128, Backbone=Llama-3.1-8B2026.04 | 11.93 | 0 | 0.15 | 3.06 | 91 | 11.82 | 137.35 | 100 | |
| Syncpoolk=128, Backbone=Llama-3.1-8B2026.04 | 11.98 | 0 | 0 | 1.01 | 33 | 5.15 | 3.41 | 100 | |
| Verification-basedk=256, Backbone=Llama-3.1-8B2026.04 | 13.54 | 0 | 0 | 3.13 | 87 | 552.64 | 10,714.87 | 100 | |
| Stepwise Verificationk=512, Backbone=Llama-3.1-8B2026.04 | 16.37 | 0 | 0.01 | 3.45 | 91 | 32.39 | 525.29 | 100 | |
| Syncpoolk=512, Backbone=Llama-3.1-8B2026.04 | 17.06 | 0 | 0 | 0.53 | 14 | 5.21 | 0.58 | 100 | |
| Discopk=512, Backbone=Llama-3.1-8B2026.04 | 17.29 | 0 | 0 | 3.5 | 91 | 5.18 | 0 | — | |
| ReTokSynck=512, Backbone=Llama-3.1-8B2026.04 | 17.29 | 0 | 0 | 3.5 | 91 | 5.27 | 1.74 | 100 | |
| Basick=32, Backbone=Llama-3.1-8B2026.04 | 19.93 | 0.96 | 9.15 | 2.72 | 96 | 7.64 | 55.92 | 100 | |
| Basick=128, Backbone=Llama-3.1-8B2026.04 | 59.05 | 1.24 | 8.77 | 4.05 | 98 | 8.75 | 75.7 | 100 | |
| Basick=512, Backbone=Llama-3.1-8B2026.04 | 219.67 | 1.36 | 7.58 | 5.72 | 98 | 21.69 | 318.73 | 100 |