Tokenization on Taiwan lexicon
1.7Weighted CostTAIDE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| TAIDEVocab.=318,0802026.07 | 1.7 | 20 | |
| PangolinTokenizerVocab.=114,8222026.07 | 2.48 | 27 | |
| Qwen 3.6Vocab.=248,0702026.07 | 2.59 | 27 | |
| Gemma 3/4Vocab.=262,1442026.07 | 2.92 | 33 | |
| Qwen 2.5/3Vocab.=151,6692026.07 | 3.08 | 47 | |
| GPT-4oVocab.=200,0192026.07 | 3.45 | 40 | |
| LLaMA 3.1Vocab.=128,2562026.07 | 3.47 | 53 | |
| VoxCPM2Vocab.=122,7532026.07 | 4.31 | 60 |