Bigram Language Modeling on Synthetic WebText initialization (val)
0.001Avg JSMixture KLs
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Mixture KLsVocab=20, Objective=Mix KLs2023.05 | 0.001 | 0 | |
| Forward KLVocab=20, Objective=For. KL2023.05 | 0.001 | 0.0002 | |
| MIXCEVocab=20, Objective=MIXCE2023.05 | 0.001 | 0.0002 | |
| Oracle MIXCEVocab=20, Objective=MIXCE*2023.05 | 0.0014 | 0.0001 | |
| Mixture KLsVocab=50, Objective=Mix KLs2023.05 | 0.0039 | 0.0003 | |
| Reverse KLVocab=20, Objective=Rev. KL2023.05 | 0.0039 | 0 | |
| Oracle MIXCEVocab=50, Objective=MIXCE*2023.05 | 0.0039 | 0.0003 | |
| MIXCEVocab=50, Objective=MIXCE2023.05 | 0.0043 | 0.0005 | |
| Forward KLVocab=50, Objective=For. KL2023.05 | 0.0043 | 0.0005 | |
| Mixture KLsVocab=100, Objective=Mix KLs2023.05 | 0.0092 | 0.0002 | |
| Jensen-ShannonVocab=50, Objective=JS2023.05 | 0.0092 | 0 | |
| Oracle MIXCEVocab=100, Objective=MIXCE*2023.05 | 0.0096 | 0.0002 | |
| MIXCEVocab=100, Objective=MIXCE2023.05 | 0.0097 | 0 | |
| Forward KLVocab=100, Objective=For. KL2023.05 | 0.0097 | 0.0003 | |
| Jensen-ShannonVocab=20, Objective=JS2023.05 | 0.0112 | 0 | |
| Reverse KLVocab=50, Objective=Rev. KL2023.05 | 0.0348 | 0.0001 | |
| Oracle MIXCEVocab=500, Objective=MIXCE*2023.05 | 0.0423 | 0.0001 | |
| Mixture KLsVocab=500, Objective=Mix KLs2023.05 | 0.0427 | 0.0001 | |
| MIXCEVocab=500, Objective=MIXCE2023.05 | 0.0453 | 0.0002 | |
| Forward KLVocab=500, Objective=For. KL2023.05 | 0.046 | 0.0002 | |
| Oracle MIXCEVocab=1000, Objective=MIXCE*2023.05 | 0.0744 | 0.0001 | |
| Mixture KLsVocab=1000, Objective=Mix KLs2023.05 | 0.075 | 0.0001 | |
| MIXCEVocab=1000, Objective=MIXCE2023.05 | 0.0794 | 0.0001 | |
| Forward KLVocab=1000, Objective=For. KL2023.05 | 0.081 | 0.0002 | |
| Reverse KLVocab=100, Objective=Rev. KL2023.05 | 0.1 | 0 | |
| Jensen-ShannonVocab=100, Objective=JS2023.05 | 0.173 | 0.0006 | |
| Jensen-ShannonVocab=500, Objective=JS2023.05 | 0.278 | 0 | |
| Jensen-ShannonVocab=1000, Objective=JS2023.05 | 0.302 | 0 | |
| Reverse KLVocab=500, Objective=Rev. KL2023.05 | 0.305 | 0 | |
| Reverse KLVocab=1000, Objective=Rev. KL2023.05 | 0.33 | 0 |