List your company
ResearchFormerly WizWand

Language Modeling on Wikipedia

9.17PerplexityOriginal
5.527230.116154.70579.2939Aug 5, 2024Nov 5, 2024Feb 6, 2025May 10, 2025Aug 11, 2025Nov 12, 2025Feb 13, 2026
Updated 5mo ago

Evaluation results

MethodLinks
OriginalBackbone=Llama3-8B2026.029.17
Linear-AcTBackbone=Llama3-8B2026.029.32
BioMamba-2.7B# Parameters=2.7B2024.089.71
CHaRS-PCTBackbone=Llama3-8B2026.029.79
CHaRSBackbone=Llama3-8B2026.029.85
OriginalBackbone=Qwen2.5-7B2026.0210.13
Linear-AcTBackbone=Qwen2.5-7B2026.0210.36
CHaRS-PCTBackbone=Qwen2.5-7B2026.0210.51
CHaRSBackbone=Qwen2.5-7B2026.0210.61
Meditron3-Gemma2-2B# Parameters=2B2024.0811.61
MIDUS-HMLTraining Stage=CPT-1B, Backbone=Llama-3.2-1B2025.1211.64
OpT-DeUSTraining Stage=CPT-1B, Backbone=Llama-3.2-1B2025.1211.72
MIDUS-HMLTraining Stage=SFT-1B, Backbone=Llama-3.2-1B2025.1211.75
OpT-DeUSTraining Stage=SFT-1B, Backbone=Llama-3.2-1B2025.1211.82
Avg-DeUSTraining Stage=CPT-1B, Backbone=Llama-3.2-1B2025.1212.04
LESATraining Stage=CPT-1B, Backbone=Llama-3.2-1B2025.1212.06
GPT2 - MModel=GPT2 - M, Bits per Token=7864322025.0512.16
Avg-DeUSTraining Stage=SFT-1B, Backbone=Llama-3.2-1B2025.1212.18
Llama ProTraining Stage=CPT-1B, Backbone=Llama-3.2-1B2025.1212.33
LESATraining Stage=SFT-1B, Backbone=Llama-3.2-1B2025.1212.37
Llama ProTraining Stage=SFT-1B, Backbone=Llama-3.2-1B2025.1212.55
BaseTraining Stage=SFT-1B, Backbone=Llama-3.2-1B2025.1213.09
BaseTraining Stage=CPT-1B, Backbone=Llama-3.2-1B2025.1213.22
SOLARTraining Stage=SFT-1B, Backbone=Llama-3.2-1B2025.1213.23
SOLARTraining Stage=CPT-1B, Backbone=Llama-3.2-1B2025.1213.41
ASTRAModel=GPT2 - M, #Groups=32, Bits per Token=7680, Compression Ratio=102.42025.0513.83
OriginalBackbone=Gemma2-2B2026.0214.4
ASTRAModel=GPT2 - M, #Groups=16, Bits per Token=3840, Compression Ratio=204.82025.0514.43
Linear-AcTBackbone=Gemma2-2B2026.0214.62
CHaRSBackbone=Gemma2-2B2026.0215.06
CHaRS-PCTBackbone=Gemma2-2B2026.0215.17
GPT2 - SModel=GPT2 - S, Bits per Token=2949122025.0515.79
ASTRAModel=GPT2 - S, #Groups=32, Bits per Token=3840, Compression Ratio=76.82025.0517.39
ASTRAModel=GPT2 - M, #Groups=1, Bits per Token=240, Compression Ratio=3276.82025.0517.86
ASTRAModel=GPT2 - S, #Groups=16, Bits per Token=1920, Compression Ratio=153.62025.0518.84
BioMamba-130M# Parameters=130M2024.0820.49
ASTRAModel=GPT2 - S, #Groups=1, Bits per Token=120, Compression Ratio=2457.62025.0521.46
Bio-Medical-Llama-3.2-1B# Parameters=1B2024.0822.54
Gemma3-1B# Parameters=∼1B, mode=fine-tuned2024.0835.01
BioGPT-Large# Parameters=1.5B2024.0835.83
BioGPT-Large-PubMedQA# Parameters=1.5B2024.0843.54
BioGPT# Parameters=347M2024.0861.31
BioMedLM# Parameters=2.7B2024.08100.24