Categories Companies Research

Loading the SOTA2 catalog…

Explore

Categories Companies Research

Contribute

Join SOTA2 Contact

SOTA2

© 2026 SOTA2

Build in Bellevue, USA

WizWand is now SOTA2 Research

Papers Benchmarks Datasets Tasks

Predicting tokenizer outperformance research benchmarks · SOTA2 Research

Predicting tokenizer outperformance

Benchmarks

Dataset NameSOTA MethodSort

Dataset NameSOTA methodMetricTrendResultsLast Updated

Pairwise Tokenizer Comparisons Average (val)

Two-stage predictive framework

0.76F1 Score

6

Feb 26, 2026

Pairwise Tokenizer Comparisons AYA 23 held-out (val)

92F1 Score

6

Feb 26, 2026

Pairwise Tokenizer Comparisons TIKTOKEN held-out (val)

Two-stage predictive framework

89F1 Score

6

Feb 26, 2026

Pairwise Tokenizer Comparisons FALCON held-out (val)

Two-stage predictive framework

83F1 Score

6

Feb 26, 2026

Pairwise Tokenizer Comparisons GPT-NEOX held-out (val)

Deviation from linear function

100F1 Score

6

Feb 26, 2026

Pairwise Tokenizer Comparisons GPT-2 held-out (val)

Two-stage predictive framework

0.67F1 Score

6

Feb 26, 2026

Pairwise Tokenizer Comparisons PHI-3-MINI held-out (val)

Deviation from linear function

76F1 Score

6

Feb 26, 2026

Popular datasets

Pairwise Tokenizer Comparisons

Follow for updates

Join SOTA2 Contact