Language Modeling on OWT Qwen-2 tokenizer (val)
21.9Validation PerplexityCCDD-MoEDiT
Evaluation Results
| Method | Links | |
|---|---|---|
| CCDD-MoEDiTTrain. toks.=131B, # params.=104.0M, Embedding=Qwen3-Embedding-0.6B2025.10 | 21.9 | |
| CCDD-MMDiTTrain. toks.=131B, # params.=124.0M, Embedding=Qwen3-Embedding-0.6B2025.10 | 26.59 | |
| CCDD-MDiTTrain. toks.=131B, # params.=92.1M, Embedding=Qwen3-Embedding-0.6B2025.10 | 29.18 | |
| MDLMTrain. toks.=131B, # params.=92.1M, Implementation=reimpl.2025.10 | 33.78 |