Zero-shot Language Evaluation on Gauntlet 20 benchmarks (test)
9.2Average Normalized AccuracyPrior-based
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Prior-basedModel Size=1.5B, Type=ours, Filtering Time=0.25 hrs2025.09 | 9.2 | 9.53 | 11.27 | 10.31 | 11.13 | 3.79 | |
| PPL-basedModel Size=1.5B, Type=model-based, Filtering Time=216 GPU hrs2025.09 | 8.22 | 9.98 | 11.91 | 7.34 | 7.91 | 3.96 | |
| DSIRModel Size=1.5B, Type=n-gram, Filtering Time=4 hrs2025.09 | 7.56 | 7.03 | 6.84 | 7.31 | 12.67 | 3.97 | |
| FastTextModel Size=1.5B, Type=classifier, Filtering Time=3.6 hrs2025.09 | 7.09 | 6.71 | 6.11 | 6.89 | 11.93 | 3.82 | |
| Prior-basedModel Size=137M, Type=ours, Filtering Time=0.25 hours2025.09 | 6.65 | 5.03 | 9.13 | 4.22 | 11.21 | 3.66 | |
| vanillaModel Size=1.5B, Type=rule-based2025.09 | 5.78 | 5.52 | 0.44 | 6.14 | 13.22 | 3.59 | |
| DSIRModel Size=137M, Type=n-gram, Filtering Time=4 hrs2025.09 | 5.6 | 5.68 | 4.93 | 1.97 | 11.6 | 3.8 | |
| FastTextModel Size=137M, Type=classifier, Filtering Time=3.6 hrs2025.09 | 5.39 | 5.12 | 4.29 | 1.74 | 12.31 | 3.49 | |
| PPL-basedModel Size=137M, Type=model-based, Filtering Time=216 GPU hrs2025.09 | 5.26 | 5.47 | 6.53 | 2.9 | 7.84 | 3.58 | |
| vanillaModel Size=137M, Type=rule-based2025.09 | 4.96 | 4.96 | 1.81 | 1.47 | 12.83 | 3.7 |