Probability Distribution Modeling on Llama 3.1 8B (val)
0.001DKLBaseline
Evaluation Results
| Method | Links | |
|---|---|---|
| BaselineFormat=bfloat16, Bit width (b)=16.00, Backbone=Llama 3.1 8B2025.05 | 0.001 | |
| Tensor RMS + CFormat=Tensor RMS + Cube Root Density, Bit width (b)=3.00, Quantization Method=direct-cast, Backbone=Llama 3.1 8B2025.05 | 0.205 | |
| Tensor RMS + SpFormat=Tensor RMS + Signmax Scaling, Bit width (b)=3.05, Quantization Method=direct-cast, Backbone=Llama 3.1 8B2025.05 | 0.503 | |
| Channel AbsmaxFormat=Channel Absmax, Bit width (b)=3.00, Quantization Method=direct-cast, Backbone=Llama 3.1 8B2025.05 | 1.075 | |
| Block AbsmaxFormat=Block Absmax, Bit width (b)=3.25, Quantization Method=direct-cast, Backbone=Llama 3.1 8B2025.05 | 1.264 | |
| Tensor AbsmaxFormat=Tensor Absmax, Bit width (b)=3.00, Quantization Method=direct-cast, Backbone=Llama 3.1 8B2025.05 | 4.577 | |
| Tensor RMSFormat=Tensor RMS, Bit width (b)=3.00, Quantization Method=direct-cast, Backbone=Llama 3.1 8B2025.05 | 9.115 |