Language Modeling on WikiText (PPL, Util. (TFLOP/s), Latency, Throughput)
17.69PPLVanilla-16L
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Vanilla-16LParams (B)=1.28, Cost (TFLOPs)=4.822026.06 | 17.69 | 53.87 | 89.47 | 22,890 | |
| CascadeFormer-A1Params (B)=1.25, Cost (TFLOPs)=4.722026.06 | 17.79 | 55.39 | 85.17 | 24,048 | |
| Vanilla-15LParams (B)=1.21, Cost (TFLOPs)=4.542026.06 | 17.84 | 54.23 | 83.81 | 24,439 | |
| CascadeFormer-A2Params (B)=1.22, Cost (TFLOPs)=4.582026.06 | 17.84 | 59.77 | 76.62 | 26,731 | |
| CascadeFormer-F1Params (B)=1.19, Cost (TFLOPs)=4.432026.06 | 17.88 | 51.16 | 86.65 | 23,635 | |
| CascadeFormer-C1Params (B)=1.16, Cost (TFLOPs)=4.332026.06 | 17.94 | 53 | 81.71 | 25,066 | |
| Vanilla-14LParams (B)=1.15, Cost (TFLOPs)=4.272026.06 | 18 | 54.68 | 78.09 | 26,227 | |
| CascadeFormer-F2Params (B)=1.09, Cost (TFLOPs)=4.052026.06 | 18.1 | 49.92 | 81.06 | 25,266 | |
| Vanilla-13LParams (B)=1.08, Cost (TFLOPs)=4.002026.06 | 18.17 | 55.18 | 72.4 | 28,287 | |
| CascadeFormer-C2Params (B)=1.03, Cost (TFLOPs)=3.812026.06 | 18.3 | 53.88 | 70.65 | 28,990 | |
| Vanilla-12LParams (B)=1.01, Cost (TFLOPs)=3.722026.06 | 18.82 | 55.74 | 66.75 | 30,683 |