Tokenization on Japanese
1,040.5Avg Tokens per SampleOpenAI o200k
Evaluation Results
| Method | Links | |
|---|---|---|
| OpenAI o200kVocabulary Size=200k2026.01 | 1,040.5 | |
| Qwen32026.01 | 935.87 | |
| A.X K12026.01 | 871.3 |
Loading the SOTA2 catalog…
| Method | Links | |
|---|---|---|
| OpenAI o200kVocabulary Size=200k2026.01 | 1,040.5 | |
| Qwen32026.01 | 935.87 | |
| A.X K12026.01 | 871.3 |