Predicting tokenizer outperformance on Pairwise Tokenizer Comparisons Average (val)
0.76F1 ScoreTwo-stage predictive framework
Evaluation results
| Method | Links | |
|---|---|---|
| Two-stage predictive frameworkIntrinsic metric=C + P + S, Predictive model=SVM (linear kernel)2025.06 | 0.76 | |
| Deviation from linear functionIntrinsic metric=POWER LAW (P), Predictive model=Logistic Regression2025.06 | 0.7 | |
| Slope of linear functionIntrinsic metric=SLOPE (S), Predictive model=Logistic Regression2025.06 | 0.55 | |
| COMPRESSIONIntrinsic metric=COMPRESSION, Predictive model=Logistic Regression2025.06 | 0.53 | |
| CARDINALITYIntrinsic metric=CARDINALITY (C), Predictive model=Logistic Regression2025.06 | 0.51 | |
| Rank-frequency AUCIntrinsic metric=AUC, Predictive model=Logistic Regression2025.06 | 0.42 |