Tokenization efficiency on Twitter
1,303Token CountCSV
Evaluation Results
| Method | Links | |
|---|---|---|
| CSVSerialization format=CSV, Tokenizer=o200k_base2026.04 | 1,303 | |
| Zen GridSerialization format=Zen Grid, Formatting options=bare_strings, Tokenizer=o200k_base2026.04 | 1,422 | |
| MarkdownSerialization format=Markdown, Tokenizer=o200k_base2026.04 | 1,430 | |
| Zen GridSerialization format=Zen Grid, Tokenizer=o200k_base2026.04 | 1,653 | |
| YAMLSerialization format=YAML, Tokenizer=o200k_base2026.04 | 1,916 | |
| JSON CompactSerialization format=JSON Compact, Tokenizer=o200k_base2026.04 | 3,673 | |
| JSON PrettySerialization format=JSON Pretty, Tokenizer=o200k_base2026.04 | 4,166 |