List your company
ResearchFormerly WizWand

Language Modeling on WikiText (held-out)

9.8Perplexity (PPL)Before finetune
7.850421.010234.1747.3298Jun 12, 2024Aug 9, 2024Oct 6, 2024Dec 4, 2024Jan 31, 2025Mar 30, 2025May 28, 2025
Updated 8mo ago

Evaluation results

MethodLinks
Before finetuneBackbone=Mistral-7B-instruct2024.069.8
Target LLMBackbone=Mistral-7B-instruct2024.069.81
ULDBackbone=Mistral-7B-instruct2024.069.95
NPO+GDBackbone=Mistral-7B-instruct2024.0614.85
NPO+KLBackbone=Mistral-7B-instruct2024.0615.44
GA+GDBackbone=Mistral-7B-instruct2024.0615.75
DPO+GDBackbone=Mistral-7B-instruct2024.0616.98
GA+KLBackbone=Mistral-7B-instruct2024.0617.59
RoPEModel size=120M, Training context length=512, Evaluation context length=up to 5122025.0518.3599
DPO+KLBackbone=Mistral-7B-instruct2024.0618.43
ALiBi SSMaxModel size=120M, Training context length=512, Evaluation context length=up to 5122025.0518.4967
ALiBiModel size=120M, Training context length=512, Evaluation context length=up to 5122025.0518.5452
BAM SSMaxModel size=120M, Training context length=512, Evaluation context length=up to 5122025.0518.6897
BAMModel size=120M, Training context length=512, Evaluation context length=up to 5122025.0518.9281
SinusoidalModel size=120M, Training context length=512, Evaluation context length=up to 5122025.0518.931
Sinusoidal SSMaxModel size=120M, Training context length=512, Evaluation context length=up to 5122025.0519.1467
Offset-NPO+KLBackbone=Mistral-7B-instruct2024.0619.43
NoPEModel size=120M, Training context length=512, Evaluation context length=up to 5122025.0519.8609
RoPE SSMaxModel size=120M, Training context length=512, Evaluation context length=up to 5122025.0520.1854
NoPE SSMaxModel size=120M, Training context length=512, Evaluation context length=up to 5122025.0520.3899
Offset-DPO+KLBackbone=Mistral-7B-instruct2024.0623.56
NPOBackbone=Mistral-7B-instruct2024.0635.71
DPOBackbone=Mistral-7B-instruct2024.0642.14
GABackbone=Mistral-7B-instruct2024.0648.13
Offset-GA+KLBackbone=Mistral-7B-instruct2024.0658.54