Tokenization efficiency on GitHub
688Token CountCSV
Evaluation results
| Method | Links | |
|---|---|---|
| CSVSerialization format=CSV, Tokenizer=o200k_base2026.04 | 688 | |
| MarkdownSerialization format=Markdown, Tokenizer=o200k_base2026.04 | 792 | |
| JSON CompactSerialization format=JSON Compact, Tokenizer=o200k_base2026.04 | 968 | |
| Zen GridSerialization format=Zen Grid, Tokenizer=o200k_base2026.04 | 968 | |
| YAMLSerialization format=YAML, Tokenizer=o200k_base2026.04 | 1,185 | |
| JSON PrettySerialization format=JSON Pretty, Tokenizer=o200k_base2026.04 | 1,388 |