Tokenization Efficiency
Benchmarks
Dataset NameSOTA methodMetricTrendResultsLast Updated
100NSL Score
25
Mar 24, 2026
10.43Mean Brahmic Fertility
11
May 29, 2026
10.97Mean Brahmic Fertility
11
May 29, 2026
1,303Token Count
7
Apr 8, 2026
408Token Count
6
Apr 8, 2026
688Token Count
6
Apr 8, 2026
1Bytes per Token
6
Mar 24, 2026
4.43Bytes per Token
6
Mar 24, 2026
4.1Bytes per Token
6
Mar 24, 2026
1.02Bytes per Token
6
Mar 24, 2026
1.02Bytes per Token
6
Mar 24, 2026
5.45Bytes per Token
6
Mar 24, 2026
4.05Bytes per Token
6
Mar 24, 2026
1.58Bytes-per-token
6
Mar 24, 2026
6.84Bytes-per-token
6
Mar 24, 2026
6.49Bytes per Token
6
Mar 24, 2026
2.81Bytes per Token
6
Mar 24, 2026
5.26Bytes per Token
6
Mar 24, 2026
5.39Bytes-per-token
6
Mar 24, 2026
5.33Bytes per Token
6
Mar 24, 2026
5.29Bytes per Token
6
Mar 24, 2026
7.37Bytes per Token
6
Mar 24, 2026
4.69Bytes per Token
6
Mar 24, 2026
4.55Bytes per Token
6
Mar 24, 2026
3.65Bytes per Token
6
Mar 24, 2026