Language Modeling on Stories (test)
0.809Bits Per ByteTransformer-1B
Evaluation results
| Method | Links | |
|---|---|---|
| Transformer-1BTokenization=subword, Context size=2048 tokens (~8192 bytes), Data trained=≈ 30B bytes2024.04 | 0.809 | |
| SpaceByte-793M+184MTokenization=byte-level, Context size=8192 bytes, Data trained=30B bytes2024.04 | 0.833 | |
| MambaByte-353MTokenization=byte-level, Context size=8192 bytes, Data trained=30B bytes2024.04 | 0.908 | |
| MegaByte-758M+262MTokenization=byte-level, Context size=8192 bytes, Data trained=80B bytes2024.04 | 0.978 | |
| Transformer-320MTokenization=byte-level, Context size=1024 bytes, Data trained=80B bytes2024.04 | 1.064 | |
| PerceiverAR-248MTokenization=byte-level, Context size=8192 bytes, Data trained=80B bytes2024.04 | 1.07 |