Semantic and linguistic knowledge evaluation on ZeroSpeech
61.9sBLiMP ScoreSmolTolk 2B
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| SmolTolk 2BParams=2.0B, Textually Pretrained=✓, Codebooks=1 × 5002026.03 | 61.9 | — | — | |
| SpeechSSM 2BParams=2.0B, Textually Pretrained=✓, Codebooks=1 × 320002026.03 | 60.9 | 55.8 | — | |
| HuBERT 25HzParams=∼357M, Textually Pretrained=✓, Codebooks=1 × 5002026.03 | 60.9 | 70.5 | 65.3 | |
| TWIST 7BParams=7.0B, Textually Pretrained=✓, Codebooks=1 × 5002026.03 | 59.2 | 74.5 | 66.3 | |
| Moshi 7BParams=7.7B, Textually Pretrained=✓, Codebooks=8 × 20482026.03 | 58.8 | 72.6 | — | |
| SpiRit LMParams=7.0B, Textually Pretrained=✓, Codebooks=1 × 5012026.03 | 58.3 | 69 | 64.2 | |
| TWIST 1.3BParams=1.3B, Textually Pretrained=✓, Codebooks=1 × 5002026.03 | 57 | 72.7 | 64.8 | |
| LLaMA-Mimi 8BParams=8.0B, Textually Pretrained=✓, Codebooks=4 × 20482026.03 | 55.1 | 68.8 | 69.5 | |
| WavSLM-2kParams=305M, Textually Pretrained=✗, Codebooks=1 × 20482026.03 | 54.6 | 63.9 | 68.3 | |
| LLaMA-Mimi 1.3BParams=1.3B, Textually Pretrained=✓, Codebooks=4 × 20482026.03 | 54.3 | 68.7 | 69 | |
| SpiRit LM Expr.Params=7.0B, Textually Pretrained=✓, Codebooks=[501, 64, 100]2026.03 | 54.2 | 65 | 69.4 | |
| DWavL-S-16Params=∼357M, Textually Pretrained=✓, Codebooks=6 × 10002026.03 | 54 | 69.1 | 69 | |
| WavSLM-4kParams=307M, Textually Pretrained=✗, Codebooks=1 × 40962026.03 | 53.9 | 63.7 | 69.5 | |
| Mimi-S-24Params=∼357M, Textually Pretrained=✓, Codebooks=8 × 20482026.03 | 52.3 | 62.2 | 64 | |
| WavSLM-65kParams=370M, Textually Pretrained=✗, Codebooks=1 × 655362026.03 | 52.1 | 61.4 | 66.5 | |
| SQ-SMA-16Params=∼357M, Textually Pretrained=✓, Codebooks=4 × 196832026.03 | 51.6 | 51.4 | 62.9 | |
| DAC-SMA-16Params=∼357M, Textually Pretrained=✓, Codebooks=8 × 10242026.03 | 51.5 | 50.7 | 60.2 | |
| WT-SMA-24Params=∼357M, Textually Pretrained=✓, Codebooks=1 × 40962026.03 | 51.2 | 54.6 | 62.6 | |
| Enc-SMA-24Params=∼357M, Textually Pretrained=✓, Codebooks=8 × 10242026.03 | 51.1 | 51.3 | 56.1 | |
| ST-S-16Params=∼357M, Textually Pretrained=✓, Codebooks=8 × 10242026.03 | 51.1 | 56.9 | 57.2 |