Next token prediction on Pre-training corpus (train)
66.4Token AccuracyLIME+1
Evaluation results
| Method | Links | ||
|---|---|---|---|
| LIME+1Size=2B, Total Parameters=2,426,626,048, LIME Parameters=+145,4082025.12 | 66.4 | 4.28 | |
| LIME+1Size=1B, Total Parameters=919,728,128, LIME Parameters=+72,7042025.12 | 64.86 | 4.73 | |
| LIME+1Size=500M, Total Parameters=495,919,104, LIME Parameters=+54,5282025.12 | 61.83 | 5.77 | |
| LIMESize=2B, Total Parameters=2,426,626,048, LIME Parameters=+145,4082025.12 | 47.95 | 12.1 | |
| BaselineSize=2B, Total Parameters=2,426,480,6402025.12 | 46.92 | 12.82 | |
| LIMESize=1B, Total Parameters=919,728,128, LIME Parameters=+72,7042025.12 | 46.21 | 14.02 | |
| BaselineSize=1B, Total Parameters=919,655,4242025.12 | 45.2 | 14.84 | |
| LIMESize=500M, Total Parameters=495,919,104, LIME Parameters=+54,5282025.12 | 42.91 | 18.5 | |
| BaselineSize=500M, Total Parameters=495,864,5762025.12 | 41.9 | 19.72 |