Language Modeling on PubMed
6.43PerplexityLoRA CE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| LoRA CEAdapter=LoRA, Method=CE, Precision=BF16, Seeds=32026.05 | 6.43 | 0.74 | |
| DoRA CEAdapter=DoRA, Method=CE, Precision=BF16, Seeds=32026.05 | 6.43 | 0.74 | |
| SineLoRA CEAdapter=SineLoRA, Method=CE, Precision=BF16, Seeds=32026.05 | 6.45 | 0.73 | |
| RandLoRA CEAdapter=RandLoRA, Method=CE, Precision=BF16, Seeds=32026.05 | 6.48 | 0.7 | |
| CEBackbone=Qwen2.5-7B-Instruct, Adapter=DoRA2026.05 | 6.49 | — | |
| CEBackbone=Qwen2.5-7B-Instruct, Adapter=LoRA2026.05 | 6.5 | — | |
| BioMamba-2.7B# Parameters=2.7B2024.08 | 6.52 | — | |
| CEBackbone=Qwen2.5-7B-Instruct, Adapter=SineLoRA2026.05 | 6.52 | — | |
| CEBackbone=Qwen2.5-7B-Instruct, Adapter=RandLoRA2026.05 | 6.55 | — | |
| LoRA CE + TMKLAdapter=LoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.05 | 6.55 | 0.63 | |
| DoRA CE + TMKLAdapter=DoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.05 | 6.55 | 0.63 | |
| SineLoRA CE + TMKLAdapter=SineLoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.05 | 6.58 | 0.6 | |
| RandLoRA CE + TMKLAdapter=RandLoRA, Method=CE + TMKL, Precision=BF16, Seeds=32026.05 | 6.6 | 0.58 | |
| Meditron3-Gemma2-2B# Parameters=2B2024.08 | 6.67 | — | |
| CE + TMKLBackbone=Qwen2.5-7B-Instruct, Adapter=DoRA2026.05 | 6.67 | — | |
| CE + TMKLBackbone=Qwen2.5-7B-Instruct, Adapter=LoRA2026.05 | 6.68 | — | |
| CE + TMKLBackbone=Qwen2.5-7B-Instruct, Adapter=SineLoRA2026.05 | 6.71 | — | |
| CE + TMKLBackbone=Qwen2.5-7B-Instruct, Adapter=RandLoRA2026.05 | 6.74 | — | |
| Base Qwen2.5-7BAdapter=None, Method=No adaptation, Precision=BF16, Seeds=32026.05 | 7.18 | — | |
| no adaptationBackbone=Qwen2.5-7B-Instruct2026.05 | 7.3 | — | |
| BioGPT-Large-PubMedQA# Parameters=1.5B2024.08 | 9.26 | — | |
| BioGPT-Large# Parameters=1.5B2024.08 | 10 | — | |
| BioMamba-130M# Parameters=130M2024.08 | 11.08 | — | |
| ARMzero-shot=true, number of parameters=1B, training tokens=300B2026.01 | 11.93 | — | |
| CARDzero-shot=true, number of parameters=1B, training tokens=300B2026.01 | 13.2 | — | |
| Bio-Medical-Llama-3.2-1B# Parameters=1B2024.08 | 14.46 | — | |
| BioGPT# Parameters=347M2024.08 | 15.09 | — | |
| MDLMzero-shot=true, number of parameters=1B, training tokens=300B2026.01 | 17.23 | — | |
| EntmaxContext Length=2K2025.06 | 17.64 | — | |
| ASEntmaxContext Length=2K2025.06 | 17.69 | — | |
| SSMaxContext Length=2K2025.06 | 17.84 | — | |
| SoftmaxContext Length=2K2025.06 | 18.05 | — | |
| ASEntmaxContext Length=1K2025.06 | 19.04 | — | |
| EntmaxContext Length=1K2025.06 | 19.1 | — | |
| SSMaxContext Length=1K2025.06 | 19.35 | — | |
| SoftmaxContext Length=1K2025.06 | 19.54 | — | |
| Gemma3-1B# Parameters=∼1B, mode=fine-tuned2024.08 | 19.9 | — | |
| BioMedLM# Parameters=2.7B2024.08 | 25.27 | — | |
| BD3LMzero-shot=true, number of parameters=1B, training tokens=300B2026.01 | 34.66 | — | |
| DID-FSize=Medium, Zero-shot=true, Alignment=FLOPs-aligned2026.03 | 38.71 | — | |
| DID-SSize=Medium, Zero-shot=true, Alignment=Steps-aligned2026.03 | 40.84 | — | |
| RADDSize=Medium, Zero-shot=true2026.03 | 41.06 | — | |
| XDLMSetting=Zero-shot2026.02 | 41.391 | — | |
| SCMDMAddZero-shot=true, Training tokens=262B + 3.25B, Pre-training dataset=OWT2026.04 | 41.72 | — | |
| SCMDMConcatZero-shot=true, Training tokens=262B + 3.25B, Pre-training dataset=OWT2026.04 | 41.72 | — | |
| MDLMModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 41.89 | — | |
| MDLMSetting=Zero-shot2026.02 | 41.981 | — | |
| GIDDSetting=Zero-shot2026.02 | 42.634 | — | |
| MDLMZero-shot=true, Training tokens=262B + 3.25B, Pre-training dataset=OWT2026.04 | 44.45 | — | |
| DuoModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 44.48 | — | |
| SEDD AbsorbModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 44.53 | — | |
| LangFlowModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 46.74 | — | |
| UDLMSetting=Zero-shot2026.02 | 47.181 | — | |
| Autoregressive TransformerModel Category=Autoregressive, Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 49.01 | — | |
| UDLMModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 50.98 | — | |
| DID-FSize=Small, Zero-shot=true, Alignment=FLOPs-aligned2026.03 | 52.89 | — | |
| DID-SSize=Small, Zero-shot=true, Alignment=Steps-aligned2026.03 | 55.02 | — | |
| SEDD UniformModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 55.89 | — | |
| RADDSize=Small, Zero-shot=true2026.03 | 56.99 | — |