Language Modeling on The Pile (Teacher Comparison & Efficiency)
4.05Inference SpeedupBudgeted LoRA
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Budgeted LoRADense budget (F)=0.02026.05 | 4.05 | 25.13 | 28.45 | 28.86 | 0.38 | 80.2 | — | |
| AC-ODM(160M) (Proxy Policy)AC Parameters=17M, LLM Parameters=160M, Total Parameters=177M, Time (s)=0.65, Model Family=Pythia2025.05 | 2.08 | — | — | — | — | — | 28,690 | |
| AC-ODM(160M) (Target Transfer)AC Parameters=17M, LLM Parameters=1B, Total Parameters=1.02B, Time (s)=2.48, Model Family=Pythia2025.05 | 2.08 | — | — | — | — | — | 12,500 | |
| Budgeted LoRADense budget (F)=0.42026.05 | 1.74 | 24.34 | 26.19 | 26.58 | 0.59 | 53.9 | — | |
| AC-ODM(410M) (Proxy Policy)AC Parameters=17M, LLM Parameters=410M, Total Parameters=427M, Time (s)=1.41, Model Family=Pythia2025.05 | 1.47 | — | — | — | — | — | 28,690 | |
| AC-ODM(410M) (Target Transfer)AC Parameters=17M, LLM Parameters=1B, Total Parameters=1.02B, Time (s)=2.48, Model Family=Pythia2025.05 | 1.47 | — | — | — | — | — | 12,010 | |
| AC-ODMAC Parameters=17M, LLM Parameters=1B, Total Parameters=1.02B, Time (s)=2.48, Model Family=Pythia2025.05 | 1.46 | — | — | — | — | — | 28,356 | |
| PiKEAC Parameters=0, LLM Parameters=1B, Total Parameters=1B, Time (s)=2.53, Model Family=Pythia2025.05 | 1.3 | — | — | — | — | — | 31,250 | |
| KD Full2026.05 | 1 | 20.88 | 18.26 | 17.86 | 1 | 0 | — | |
| KD LoRA2026.05 | 1 | 24.32 | 26.06 | 26.5 | 0.91 | 0 | — | |
| ODMAC Parameters=0, LLM Parameters=1B, Total Parameters=1B, Time (s)=2.47, Model Family=Pythia2025.05 | 1 | — | — | — | — | — | 41,667 |