Language Modeling and Reasoning on BigBench Composite (Lamb, SQuAD, CoQA, BBH, LSAT, LangID)
24Avg ScoreKromHC
Evaluation Results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| KromHCD (Transformer blocks)=12, n (Residual streams)=42026.01 | 24 | 30.4 | 8.2 | 15.4 | 40.4 | 44.6 | 11.9 | 13.6 | 26.1 | 25 | |
| ResidualD (Transformer blocks)=12, n (Residual streams)=42026.01 | 23.7 | 29.2 | 10.8 | 13.8 | 39.2 | 38.8 | 12.9 | 15.8 | 27.8 | 25.4 | |
| mHC-liteD (Transformer blocks)=12, n (Residual streams)=42026.01 | 23.3 | 30 | 8.4 | 14.2 | 36.6 | 42.6 | 14.76 | 10 | 27 | 26.2 | |
| mHCD (Transformer blocks)=12, n (Residual streams)=42026.01 | 22.9 | 31.6 | 5.8 | 13 | 39.6 | 42 | 16.7 | 13 | 20.4 | 24 | |
| KromHCD (Transformer blocks)=6, n (Residual streams)=42026.01 | 19.5 | 19 | 0.2 | 5.8 | 14 | 40.8 | 8.6 | 11.4 | 27.8 | 27.8 | |
| mHC-liteD (Transformer blocks)=6, n (Residual streams)=42026.01 | 18.8 | 19.6 | 0.4 | 4.6 | 19.4 | 38.2 | 5.7 | 6.4 | 29.6 | 28 | |
| ResidualD (Transformer blocks)=6, n (Residual streams)=42026.01 | 18.1 | 18.6 | 0 | 5.6 | 20 | 40.8 | 9.1 | 4.6 | 23.5 | 23.4 | |
| mHCD (Transformer blocks)=6, n (Residual streams)=42026.01 | 17.3 | 17.4 | 0 | 4.8 | 10.6 | 42 | 5.2 | 9.2 | 23.5 | 26 |