Tokenization
Benchmarks
DatasetSOTA methodMetricTrendResultsLast updated
1.9NSL Score
10
Mar 24, 2026
115Speedup Ratio (GPU-O vs CPU-BPE)
5
Mar 4, 2026
3.64Compression Ratio (CR)
4
Apr 16, 2026
1,699,970Total Tokens
4
Mar 9, 2026
2.38Russian (ru)
4
Feb 26, 2026
6,623Token Count (M)
3
May 29, 2026
599Token Count (M)
3
May 29, 2026
684Token Count (M)
3
May 29, 2026
210Token Count
3
May 29, 2026
228Token Count (M)
3
May 29, 2026
529,000,000Token Count
3
May 29, 2026
518Token Count
3
May 29, 2026
313Token Count (M)
3
May 29, 2026
1,231,000,000Token Count (M)
3
May 29, 2026
605Token Count (M)
3
May 29, 2026
1,638Token Count (M)
3
May 29, 2026
100Token Count (M)
3
May 29, 2026
1,694.26Average Tokens per Sample
3
Feb 26, 2026
1,102.41Average Tokens per Sample
3
Feb 26, 2026
1,040.5Avg Tokens per Sample
3
Feb 26, 2026
914.05Average Tokens per Sample
3
Feb 26, 2026
6,192.77Average Tokens per Sample
3
Feb 26, 2026
794.79Avg Tokens per Sample
3
Feb 26, 2026
1,297.56Avg Tokens per Sample
3
Feb 26, 2026
5,520.06Avg Tokens per Sample
3
Feb 26, 2026