Tokenization on Punjabi (pa) (NSL score)
2.45NSL ScoreLLaMA-3.2-1B
Evaluation Results
| Method | Links | |
|---|---|---|
| LLaMA-3.2-1BReference Model=Base LLaMA-42025.11 | 2.45 | |
| Qwen-3Reference Model=Base LLaMA-42025.11 | 2.37 | |
| DeepSeek-R1Reference Model=Base LLaMA-42025.11 | 1.41 | |
| LLaMA-4Reference Model=Base LLaMA-42025.11 | 1 | |
| Mistral-NemoReference Model=Base LLaMA-42025.11 | 0.98 | |
| Gemma-3Reference Model=Base LLaMA-42025.11 | 0.89 | |
| GPT-OSSReference Model=Base LLaMA-42025.11 | 0.85 | |
| SarvamReference Model=Base LLaMA-42025.11 | 0.54 | |
| SutraReference Model=Base LLaMA-42025.11 | 0.51 | |
| MUTANT-IndicReference Model=Base LLaMA-42025.11 | 0.45 |