Language Modeling on Lambada-O (Accuracy)
26.19AccuracyADAPT-BM25
Evaluation results
| Method | Links | |
|---|---|---|
| ADAPT-BM25Backbone=TinyLlama-120M, Training Budget=50B tokens, FLOPs overhead=≪ 1.0 × 10^142026.04 | 26.19 | |
| RegMixBackbone=TinyLlama-120M, Training Budget=50B tokens, FLOPs overhead=3.072 × 10^182026.04 | 24.82 | |
| UniformBackbone=TinyLlama-120M, Training Budget=50B tokens, FLOPs overhead=02026.04 | 24.68 | |
| ADAPTBackbone=TinyLlama-120M, Training Budget=50B tokens, FLOPs overhead=≪ 1.1 × 10^152026.04 | 24.63 | |
| LinUpperBackbone=TinyLlama-120M, Training Budget=50B tokens, FLOPs overhead=02026.04 | 23.64 | |
| DoReMiBackbone=TinyLlama-120M, Training Budget=50B tokens, FLOPs overhead=4.92 × 10^192026.04 | 22.38 |