Language Modeling on Medical (Med)
0.1PPL Change (%) vs BaselineLoRA
Evaluation Results
| Method | Links | |
|---|---|---|
| LoRABackbone=Mistral-7B, Learning Rate=10⁻³, Tokens per domain=200K, Training Stage=+Med2026.04 | 0.1 | |
| GAINBackbone=Mistral-7B, Learning Rate=10⁻³, Tokens per domain=200K, Training Stage=+HS2026.04 | 0.8 | |
| GAINBackbone=Mistral-7B, Learning Rate=10⁻³, Tokens per domain=200K, Training Stage=+SST2026.04 | 0.9 | |
| GAINBackbone=Mistral-7B, Learning Rate=10⁻³, Tokens per domain=200K, Training Stage=+LB2026.04 | 1 | |
| GAINBackbone=Mistral-7B, Learning Rate=10⁻³, Tokens per domain=200K, Training Stage=+Fin2026.04 | 1.1 | |
| GAINBackbone=Mistral-7B, Learning Rate=10⁻³, Tokens per domain=200K, Training Stage=+PG2026.04 | 1.1 | |
| GAINBackbone=Mistral-7B, Learning Rate=10⁻³, Tokens per domain=200K, Training Stage=+MdQ2026.04 | 1.1 | |
| GAINBackbone=Mistral-7B, Learning Rate=10⁻³, Tokens per domain=200K, Training Stage=+Med2026.04 | 1.2 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+SST2026.04 | 2.1 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+LB2026.04 | 2.2 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+HS2026.04 | 2.2 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+MdQ2026.04 | 2.4 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+PG2026.04 | 2.8 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+Med2026.04 | 2.9 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+Fin2026.04 | 2.9 | |
| GAINBackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+WT2026.04 | 3 | |
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+PG2026.04 | 4.7 | |
| LoRABackbone=Mistral-7B, Learning Rate=10⁻³, Tokens per domain=200K, Training Stage=+Fin2026.04 | 6.3 | |
| LoRABackbone=Mistral-7B, Learning Rate=10⁻³, Tokens per domain=200K, Training Stage=+PG2026.04 | 6.7 | |
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+Fin2026.04 | 7.2 | |
| LoRABackbone=Mistral-7B, Learning Rate=10⁻³, Tokens per domain=200K, Training Stage=+MdQ2026.04 | 9.7 | |
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+Med2026.04 | 9.8 | |
| LoRABackbone=Mistral-7B, Learning Rate=10⁻³, Tokens per domain=200K, Training Stage=+LB2026.04 | 11 | |
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+LB2026.04 | 14.5 | |
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+MdQ2026.04 | 15 | |
| LoRABackbone=Mistral-7B, Learning Rate=10⁻³, Tokens per domain=200K, Training Stage=+SST2026.04 | 20 | |
| LoRABackbone=Mistral-7B, Learning Rate=10⁻³, Tokens per domain=200K, Training Stage=+HS2026.04 | 21.8 | |
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+HS2026.04 | 22.6 | |
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+SST2026.04 | 30.7 | |
| LoRABackbone=GPT-2 Large (774M), Learning rate=10^-3, Training sequence stage=+WT2026.04 | 32.5 |