Language Modeling on RedPajama
26.756PerplexityFP16
Evaluation results
| Method | Links | |
|---|---|---|
| FP16Model=Llama-3.2-1B2025.11 | 26.756 | |
| IntAttentionModel=Llama-3.2-1B2025.11 | 28.496 | |
| Quant-OnlyModel=Llama-3.2-1B2025.11 | 32.189 | |
| sHCModel Scale=L, Zero-shot evaluation=true, Training Dataset=FineWeb-Edu2026.03 | 1,126.3 | |
| mHC-liteModel Scale=L, Zero-shot evaluation=true, Training Dataset=FineWeb-Edu2026.03 | 1,216.4 | |
| mHCModel Scale=L, Zero-shot evaluation=true, Training Dataset=FineWeb-Edu2026.03 | 1,288.4 | |
| HCModel Scale=L, Zero-shot evaluation=true, Training Dataset=FineWeb-Edu2026.03 | 1,517.8 | |
| RCModel Scale=L, Zero-shot evaluation=true, Training Dataset=FineWeb-Edu2026.03 | 1,725.9 | |
| mHC-liteModel Scale=M, Zero-shot evaluation=true, Training Dataset=FineWeb-Edu2026.03 | 2,458.3 | |
| mHCModel Scale=M, Zero-shot evaluation=true, Training Dataset=FineWeb-Edu2026.03 | 2,757.8 | |
| sHCModel Scale=M, Zero-shot evaluation=true, Training Dataset=FineWeb-Edu2026.03 | 2,785.6 | |
| HCModel Scale=M, Zero-shot evaluation=true, Training Dataset=FineWeb-Edu2026.03 | 2,890.2 | |
| RCModel Scale=M, Zero-shot evaluation=true, Training Dataset=FineWeb-Edu2026.03 | 3,152.8 |