Language Modeling on Shakespeare (CE, CER, WER)
0.16Cross-EntropyQISA
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| QISAEmbedding Size=16, Number of Heads=1, Pretraining Epochs=2, Batch Size=10242026.02 | 0.16 | 4 | 0.25 | |
| QISA-AEmbedding Size=16, Number of Heads=1, Number of Layers=1, Pretraining Epochs=2, Batch Size=10242026.02 | 0.16 | 4 | 0.25 | |
| QISA-AEmbedding Size=16, Number of Heads=1, Number of Layers=3, Pretraining Epochs=2, Batch Size=10242026.02 | 0.16 | 4 | 0.25 | |
| QISAEmbedding Size=16, Number of Heads=4, Pretraining Epochs=2, Batch Size=10242026.02 | 0.16 | 4 | 0.25 | |
| QISA-AEmbedding Size=16, Number of Heads=4, Number of Layers=1, Pretraining Epochs=2, Batch Size=10242026.02 | 0.16 | 4 | 0.25 | |
| QISA-AEmbedding Size=16, Number of Heads=4, Number of Layers=3, Pretraining Epochs=2, Batch Size=10242026.02 | 0.16 | 4 | 0.25 | |
| QISA-AEmbedding Size=16, Number of Heads=1, Number of Layers=2, Pretraining Epochs=2, Batch Size=10242026.02 | 0.17 | 4 | 0.25 | |
| QISA-AEmbedding Size=16, Number of Heads=4, Number of Layers=2, Pretraining Epochs=2, Batch Size=10242026.02 | 0.17 | 4 | 0.25 | |
| QSANNv2Embedding Size=16, Number of Heads=4, Number of Layers=2, Pretraining Epochs=2, Batch Size=10242026.02 | 0.17 | 5 | 0.26 | |
| QSANNv1Embedding Size=16, Number of Heads=4, Number of Layers=2, Pretraining Epochs=2, Batch Size=10242026.02 | 0.18 | 5 | 0.28 | |
| QSANNv1Embedding Size=16, Number of Heads=4, Number of Layers=3, Pretraining Epochs=2, Batch Size=10242026.02 | 0.18 | 5 | 0.26 | |
| QSANNv2Embedding Size=16, Number of Heads=4, Number of Layers=3, Pretraining Epochs=2, Batch Size=10242026.02 | 0.18 | 5 | 0.26 | |
| QSANNEmbedding Size=16, Number of Heads=4, Number of Layers=1, Pretraining Epochs=2, Batch Size=10242026.02 | 0.19 | 5 | 0.28 | |
| QSANNEmbedding Size=16, Number of Heads=4, Number of Layers=2, Pretraining Epochs=2, Batch Size=10242026.02 | 0.19 | 5 | 0.28 | |
| QSANNv1Embedding Size=16, Number of Heads=4, Number of Layers=1, Pretraining Epochs=2, Batch Size=10242026.02 | 0.19 | 5 | 0.27 | |
| QSANNv2Embedding Size=16, Number of Heads=4, Number of Layers=1, Pretraining Epochs=2, Batch Size=10242026.02 | 0.19 | 5 | 0.27 | |
| QSANNEmbedding Size=16, Number of Heads=4, Number of Layers=3, Pretraining Epochs=2, Batch Size=10242026.02 | 0.2 | 5 | 0.28 | |
| QSANNEmbedding Size=16, Number of Heads=1, Number of Layers=2, Pretraining Epochs=2, Batch Size=10242026.02 | 0.21 | 6 | 0.3 | |
| QSANNv2Embedding Size=16, Number of Heads=1, Number of Layers=2, Pretraining Epochs=2, Batch Size=10242026.02 | 0.21 | 6 | 0.31 | |
| QSANNEmbedding Size=16, Number of Heads=1, Number of Layers=1, Pretraining Epochs=2, Batch Size=10242026.02 | 0.22 | 6 | 0.31 | |
| QSANNEmbedding Size=16, Number of Heads=1, Number of Layers=3, Pretraining Epochs=2, Batch Size=10242026.02 | 0.22 | 7 | 0.32 | |
| QSANNv1Embedding Size=16, Number of Heads=1, Number of Layers=1, Pretraining Epochs=2, Batch Size=10242026.02 | 0.22 | 6 | 0.31 | |
| QSANNv1Embedding Size=16, Number of Heads=1, Number of Layers=3, Pretraining Epochs=2, Batch Size=10242026.02 | 0.22 | 6 | 0.3 | |
| QSANNv2Embedding Size=16, Number of Heads=1, Number of Layers=1, Pretraining Epochs=2, Batch Size=10242026.02 | 0.22 | 6 | 0.3 | |
| QSANNv2Embedding Size=16, Number of Heads=1, Number of Layers=3, Pretraining Epochs=2, Batch Size=10242026.02 | 0.22 | 7 | 0.29 | |
| QSANNv1Embedding Size=16, Number of Heads=1, Number of Layers=2, Pretraining Epochs=2, Batch Size=10242026.02 | 0.23 | 7 | 0.32 | |
| CSAEmbedding Size=16, Number of Heads=1, Pretraining Epochs=2, Batch Size=10242026.02 | 2.16 | 62 | 1.17 | |
| CSAEmbedding Size=16, Number of Heads=4, Pretraining Epochs=2, Batch Size=10242026.02 | 2.16 | 62 | 1.18 | |
| QISA-AEmbedding Size=4, Number of Heads=1, Number of Layers=1, Pretraining Epochs=2, Batch Size=10242026.02 | 2.27 | 68 | 1.06 | |
| QISA-AEmbedding Size=4, Number of Heads=1, Number of Layers=3, Pretraining Epochs=2, Batch Size=10242026.02 | 2.27 | 67.9 | 1.06 | |
| QISA-AEmbedding Size=4, Number of Heads=1, Number of Layers=2, Pretraining Epochs=2, Batch Size=10242026.02 | 2.28 | 67.9 | 1.05 | |
| QSANNv2Embedding Size=4, Number of Heads=1, Number of Layers=2, Pretraining Epochs=2, Batch Size=10242026.02 | 2.28 | 67 | 1.5 | |
| QSANNv2Embedding Size=4, Number of Heads=1, Number of Layers=1, Pretraining Epochs=2, Batch Size=10242026.02 | 2.29 | 68 | 1.49 | |
| QISAEmbedding Size=4, Number of Heads=1, Pretraining Epochs=2, Batch Size=10242026.02 | 2.3 | 68 | 1.02 | |
| QSANNv2Embedding Size=4, Number of Heads=1, Number of Layers=3, Pretraining Epochs=2, Batch Size=10242026.02 | 2.3 | 69 | 1.48 | |
| QSANNEmbedding Size=4, Number of Heads=1, Number of Layers=2, Pretraining Epochs=2, Batch Size=10242026.02 | 2.33 | 68 | 1.33 | |
| QSANNEmbedding Size=4, Number of Heads=1, Number of Layers=1, Pretraining Epochs=2, Batch Size=10242026.02 | 2.34 | 69 | 1.34 | |
| QSANNv1Embedding Size=4, Number of Heads=1, Number of Layers=2, Pretraining Epochs=2, Batch Size=10242026.02 | 2.34 | 71 | 1.34 | |
| QSANNEmbedding Size=4, Number of Heads=1, Number of Layers=3, Pretraining Epochs=2, Batch Size=10242026.02 | 2.35 | 70 | 1.35 | |
| QSANNv1Embedding Size=4, Number of Heads=1, Number of Layers=1, Pretraining Epochs=2, Batch Size=10242026.02 | 2.35 | 70 | 1.33 | |
| QSANNv1Embedding Size=4, Number of Heads=1, Number of Layers=3, Pretraining Epochs=2, Batch Size=10242026.02 | 2.36 | 69 | 1.32 | |
| CSAEmbedding Size=4, Number of Heads=1, Pretraining Epochs=2, Batch Size=10242026.02 | 2.72 | 74 | 1.34 |