Language Modeling on C4 (Entropy, Perplexity)
1EntropyBBQ
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| BBQParams=95M, Tokens=3B, Full Perplexity=24.75, Bits=12026.03 | 1 | 49.22 | |
| QuESTParams=95M, Tokens=3B, Full Perplexity=24.75, Bits=12026.03 | 1 | 67.78 | |
| BBQParams=125M, Tokens=5B, Full Perplexity=21.51, Bits=12026.03 | 1 | 44.58 | |
| QuESTParams=125M, Tokens=5B, Full Perplexity=21.51, Bits=12026.03 | 1 | 72.54 | |
| BBQParams=200M, Tokens=10B, Full Perplexity=17.93, Bits=12026.03 | 1 | 38.27 | |
| QuESTParams=200M, Tokens=10B, Full Perplexity=17.93, Bits=12026.03 | 1 | 52.37 | |
| LSQParams=200M, Tokens=10B, Full Perplexity=17.93, Bits=22026.03 | 1.63 | 78.19 | |
| LSQParams=95M, Tokens=3B, Full Perplexity=24.75, Bits=22026.03 | 1.69 | 36.58 | |
| LSQParams=125M, Tokens=5B, Full Perplexity=21.51, Bits=22026.03 | 1.81 | 31.42 | |
| QuESTParams=95M, Tokens=3B, Full Perplexity=24.75, Bits=22026.03 | 1.92 | 35.58 | |
| QuESTParams=125M, Tokens=5B, Full Perplexity=21.51, Bits=22026.03 | 1.93 | 31.32 | |
| QuESTParams=200M, Tokens=10B, Full Perplexity=17.93, Bits=22026.03 | 1.93 | 25.46 | |
| QuESTParams=300M, Tokens=20B, Full Perplexity=15.43, Bits=22026.03 | 1.93 | 21.53 | |
| BBQParams=95M, Tokens=3B, Full Perplexity=24.75, Bits=22026.03 | 1.97 | 31.34 | |
| BBQParams=125M, Tokens=5B, Full Perplexity=21.51, Bits=22026.03 | 1.98 | 27.34 | |
| BBQParams=200M, Tokens=10B, Full Perplexity=17.93, Bits=22026.03 | 1.98 | 23.08 | |
| BBQParams=300M, Tokens=20B, Full Perplexity=15.43, Bits=22026.03 | 1.98 | 19.75 | |
| LSQParams=200M, Tokens=10B, Full Perplexity=17.93, Bits=32026.03 | 2.5 | 140.9 | |
| LSQParams=200M, Tokens=10B, Full Perplexity=17.93, Bits=42026.03 | 2.73 | 1,778 | |
| LSQParams=95M, Tokens=3B, Full Perplexity=24.75, Bits=32026.03 | 2.74 | 30.27 | |
| LSQParams=125M, Tokens=5B, Full Perplexity=21.51, Bits=32026.03 | 2.74 | 26.28 | |
| QuESTParams=95M, Tokens=3B, Full Perplexity=24.75, Bits=32026.03 | 2.78 | 29.04 | |
| QuESTParams=125M, Tokens=5B, Full Perplexity=21.51, Bits=32026.03 | 2.78 | 25.21 | |
| QuESTParams=200M, Tokens=10B, Full Perplexity=17.93, Bits=32026.03 | 2.78 | 20.82 | |
| QuESTParams=300M, Tokens=20B, Full Perplexity=15.43, Bits=32026.03 | 2.78 | 17.67 | |
| BBQParams=95M, Tokens=3B, Full Perplexity=24.75, Bits=32026.03 | 2.96 | 26.55 | |
| BBQParams=125M, Tokens=5B, Full Perplexity=21.51, Bits=32026.03 | 2.96 | 23.22 | |
| BBQParams=200M, Tokens=10B, Full Perplexity=17.93, Bits=32026.03 | 2.96 | 19.74 | |
| BBQParams=300M, Tokens=20B, Full Perplexity=15.43, Bits=32026.03 | 2.96 | 16.9 | |
| LSQParams=95M, Tokens=3B, Full Perplexity=24.75, Bits=42026.03 | 3.59 | 27.46 | |
| LSQParams=125M, Tokens=5B, Full Perplexity=21.51, Bits=42026.03 | 3.6 | 23.77 | |
| QuESTParams=95M, Tokens=3B, Full Perplexity=24.75, Bits=42026.03 | 3.61 | 26.37 | |
| QuESTParams=125M, Tokens=5B, Full Perplexity=21.51, Bits=42026.03 | 3.61 | 22.98 | |
| QuESTParams=200M, Tokens=10B, Full Perplexity=17.93, Bits=42026.03 | 3.61 | 19.06 | |
| QuESTParams=300M, Tokens=20B, Full Perplexity=15.43, Bits=42026.03 | 3.61 | 16.26 | |
| BBQParams=95M, Tokens=3B, Full Perplexity=24.75, Bits=42026.03 | 3.93 | 25.51 | |
| BBQParams=125M, Tokens=5B, Full Perplexity=21.51, Bits=42026.03 | 3.93 | 22.15 | |
| BBQParams=200M, Tokens=10B, Full Perplexity=17.93, Bits=42026.03 | 3.93 | 18.79 | |
| BBQParams=300M, Tokens=20B, Full Perplexity=15.43, Bits=42026.03 | 3.93 | 16.1 |