Language Modeling on The Pile deduplicated (val)
7.14PerplexitySPIRALFORMER-L
Evaluation Results
| Method | Links | |
|---|---|---|
| SPIRALFORMER-LModel Scale=1.4B, Layer Config=8+8×{1/16,1/8,1/4,1/2}+8, Topology=MeSH, PARAMS (M)=1423.2 / 1208.8, FLOPS (1E12)=13.132026.02 | 7.14 | |
| BASELINE (PYTHIA)Model Scale=1.4B, Layer Config=24 LAYERS, Topology=MeSH, PARAMS (M)=1423.1 / 1208.7, FLOPS (1E12)=14.082026.02 | 7.26 | |
| SPIRALFORMER-BModel Scale=1.4B, Layer Config=4+8×{1/8,1/4,1/2,1}+4, Topology=MeSH, PARAMS (M)=1020.4 / 805.9, FLOPS (1E12)=12.922026.02 | 7.3 | |
| LOOPEDFORMERModel Scale=1.4B, Layer Config=4+8×{1,1}+4, Topology=MeSH, PARAMS (M)=1020.2 / 805.8, FLOPS (1E12)=14.082026.02 | 7.39 | |
| BASELINE (PYTHIA)Model Scale=1.4B, Layer Config=24 LAYERS, Topology=Standard, PARAMS (M)=1423.0 / 1208.6, FLOPS (1E12)=14.082026.02 | 7.44 | |
| LOOPEDFORMERModel Scale=1.4B, Layer Config=4+8×{1,1}+4, Topology=Anchor, PARAMS (M)=1020.2 / 805.7, FLOPS (1E12)=14.082026.02 | 7.51 | |
| SPIRALFORMER-LModel Scale=1B, Layer Config=5+6×{1/16,1/8,1/4,1/2}+5, Topology=MeSH, PARAMS (M)=1020.4 / 805.9, FLOPS (1E12)=8.962026.02 | 7.64 | |
| SPIRALFORMER-BModel Scale=1B, Layer Config=3+5×{1/8,1/4,1/2,1}+3, Topology=MeSH, PARAMS (M)=768.6 / 554.1, FLOPS (1E12)=8.952026.02 | 7.8 | |
| LOOPEDFORMERModel Scale=1B, Layer Config=3+5×{1,1}+3, Topology=MeSH, PARAMS (M)=768.4 / 554.0, FLOPS (1E12)=9.672026.02 | 7.9 | |
| BASELINE (PYTHIA)Model Scale=1B, Layer Config=16 LAYERS, Topology=Standard, PARAMS (M)=1020.2 / 805.7, FLOPS (1E12)=9.672026.02 | 7.96 | |
| LOOPEDFORMERModel Scale=1B, Layer Config=3+5×{1,1}+3, Topology=Anchor, PARAMS (M)=768.4 / 553.9, FLOPS (1E12)=9.672026.02 | 8.1 | |
| SPIRALFORMER-LModel Scale=410M, Layer Config=8+8×{1/16,1/8,1/4,1/2}+8, Topology=MeSH, PARAMS (M)=407.5 / 302.4, FLOPS (1E12)=4.162026.02 | 8.73 | |
| SPIRALFORMER-BModel Scale=410M, Layer Config=4+8×{1/8,1/4,1/2,1}+4, Topology=MeSH, PARAMS (M)=306.8 / 201.6, FLOPS (1E12)=4.112026.02 | 9 | |
| BASELINE (PYTHIA)Model Scale=410M, Layer Config=24 LAYERS, Topology=Standard, PARAMS (M)=407.4 / 302.3, FLOPS (1E12)=4.592026.02 | 9.07 | |
| LOOPEDFORMERModel Scale=410M, Layer Config=4+8×{1,1}+4, Topology=MeSH, PARAMS (M)=306.7 / 201.6, FLOPS (1E12)=4.592026.02 | 9.09 | |
| SPIRALFORMER-BModel Scale=410M, Layer Config=4+8×{1/8,1/4,1/2,1}+4, Topology=Anchor, PARAMS (M)=306.7 / 201.6, FLOPS (1E12)=4.12026.02 | 9.13 | |
| LOOPEDFORMERModel Scale=410M, Layer Config=4+8×{1,1}+4, Topology=Anchor, PARAMS (M)=306.7 / 201.5, FLOPS (1E12)=4.592026.02 | 9.19 | |
| SPIRALFORMER-LModel Scale=160M, Layer Config=4+4×{1/16,1/8,1/4,1/2}+4, Topology=MeSH, PARAMS (M)=163.6 / 85.1, FLOPS (1E12)=1.492026.02 | 10.94 | |
| SPIRALFORMER-BModel Scale=160M, Layer Config=2+4×{1/8,1/4,1/2,1}+2, Topology=MeSH, PARAMS (M)=135.2 / 56.8, FLOPS (1E12)=1.482026.02 | 11.29 | |
| BASELINE (PYTHIA)Model Scale=160M, Layer Config=12 LAYERS, Topology=Standard, PARAMS (M)=163.5 / 85.1, FLOPS (1E12)=1.652026.02 | 11.31 | |
| LOOPEDFORMERModel Scale=160M, Layer Config=2+4×{1,1}+2, Topology=MeSH, PARAMS (M)=135.2 / 56.7, FLOPS (1E12)=1.652026.02 | 11.37 | |
| LOOPEDFORMERModel Scale=160M, Layer Config=2+4×{1,1}+2, Topology=Anchor, PARAMS (M)=135.2 / 56.7, FLOPS (1E12)=1.652026.02 | 11.63 |