Language Modeling on Wikipedia
9.17PerplexityOriginal
Evaluation Results
| Method | Links | |
|---|---|---|
| OriginalBackbone=Llama3-8B2026.02 | 9.17 | |
| Linear-AcTBackbone=Llama3-8B2026.02 | 9.32 | |
| BioMamba-2.7B# Parameters=2.7B2024.08 | 9.71 | |
| CHaRS-PCTBackbone=Llama3-8B2026.02 | 9.79 | |
| CHaRSBackbone=Llama3-8B2026.02 | 9.85 | |
| OriginalBackbone=Qwen2.5-7B2026.02 | 10.13 | |
| Linear-AcTBackbone=Qwen2.5-7B2026.02 | 10.36 | |
| CHaRS-PCTBackbone=Qwen2.5-7B2026.02 | 10.51 | |
| CHaRSBackbone=Qwen2.5-7B2026.02 | 10.61 | |
| Meditron3-Gemma2-2B# Parameters=2B2024.08 | 11.61 | |
| MIDUS-HMLTraining Stage=CPT-1B, Backbone=Llama-3.2-1B2025.12 | 11.64 | |
| OpT-DeUSTraining Stage=CPT-1B, Backbone=Llama-3.2-1B2025.12 | 11.72 | |
| MIDUS-HMLTraining Stage=SFT-1B, Backbone=Llama-3.2-1B2025.12 | 11.75 | |
| OpT-DeUSTraining Stage=SFT-1B, Backbone=Llama-3.2-1B2025.12 | 11.82 | |
| Avg-DeUSTraining Stage=CPT-1B, Backbone=Llama-3.2-1B2025.12 | 12.04 | |
| LESATraining Stage=CPT-1B, Backbone=Llama-3.2-1B2025.12 | 12.06 | |
| GPT2 - MModel=GPT2 - M, Bits per Token=7864322025.05 | 12.16 | |
| Avg-DeUSTraining Stage=SFT-1B, Backbone=Llama-3.2-1B2025.12 | 12.18 | |
| Llama ProTraining Stage=CPT-1B, Backbone=Llama-3.2-1B2025.12 | 12.33 | |
| LESATraining Stage=SFT-1B, Backbone=Llama-3.2-1B2025.12 | 12.37 | |
| Llama ProTraining Stage=SFT-1B, Backbone=Llama-3.2-1B2025.12 | 12.55 | |
| BaseTraining Stage=SFT-1B, Backbone=Llama-3.2-1B2025.12 | 13.09 | |
| BaseTraining Stage=CPT-1B, Backbone=Llama-3.2-1B2025.12 | 13.22 | |
| SOLARTraining Stage=SFT-1B, Backbone=Llama-3.2-1B2025.12 | 13.23 | |
| SOLARTraining Stage=CPT-1B, Backbone=Llama-3.2-1B2025.12 | 13.41 | |
| ASTRAModel=GPT2 - M, #Groups=32, Bits per Token=7680, Compression Ratio=102.42025.05 | 13.83 | |
| OriginalBackbone=Gemma2-2B2026.02 | 14.4 | |
| ASTRAModel=GPT2 - M, #Groups=16, Bits per Token=3840, Compression Ratio=204.82025.05 | 14.43 | |
| Linear-AcTBackbone=Gemma2-2B2026.02 | 14.62 | |
| CHaRSBackbone=Gemma2-2B2026.02 | 15.06 | |
| CHaRS-PCTBackbone=Gemma2-2B2026.02 | 15.17 | |
| GPT2 - SModel=GPT2 - S, Bits per Token=2949122025.05 | 15.79 | |
| ASTRAModel=GPT2 - S, #Groups=32, Bits per Token=3840, Compression Ratio=76.82025.05 | 17.39 | |
| ASTRAModel=GPT2 - M, #Groups=1, Bits per Token=240, Compression Ratio=3276.82025.05 | 17.86 | |
| ASTRAModel=GPT2 - S, #Groups=16, Bits per Token=1920, Compression Ratio=153.62025.05 | 18.84 | |
| BioMamba-130M# Parameters=130M2024.08 | 20.49 | |
| ASTRAModel=GPT2 - S, #Groups=1, Bits per Token=120, Compression Ratio=2457.62025.05 | 21.46 | |
| Bio-Medical-Llama-3.2-1B# Parameters=1B2024.08 | 22.54 | |
| Gemma3-1B# Parameters=∼1B, mode=fine-tuned2024.08 | 35.01 | |
| BioGPT-Large# Parameters=1.5B2024.08 | 35.83 | |
| BioGPT-Large-PubMedQA# Parameters=1.5B2024.08 | 43.54 | |
| BioGPT# Parameters=347M2024.08 | 61.31 | |
| BioMedLM# Parameters=2.7B2024.08 | 100.24 |