Language Modeling on WikiText (held-out)
9.8Perplexity (PPL)Before finetune
Evaluation Results
| Method | Links | |
|---|---|---|
| Before finetuneBackbone=Mistral-7B-instruct2024.06 | 9.8 | |
| Target LLMBackbone=Mistral-7B-instruct2024.06 | 9.81 | |
| ULDBackbone=Mistral-7B-instruct2024.06 | 9.95 | |
| NPO+GDBackbone=Mistral-7B-instruct2024.06 | 14.85 | |
| NPO+KLBackbone=Mistral-7B-instruct2024.06 | 15.44 | |
| GA+GDBackbone=Mistral-7B-instruct2024.06 | 15.75 | |
| DPO+GDBackbone=Mistral-7B-instruct2024.06 | 16.98 | |
| GA+KLBackbone=Mistral-7B-instruct2024.06 | 17.59 | |
| RoPEModel size=120M, Training context length=512, Evaluation context length=up to 5122025.05 | 18.3599 | |
| DPO+KLBackbone=Mistral-7B-instruct2024.06 | 18.43 | |
| ALiBi SSMaxModel size=120M, Training context length=512, Evaluation context length=up to 5122025.05 | 18.4967 | |
| ALiBiModel size=120M, Training context length=512, Evaluation context length=up to 5122025.05 | 18.5452 | |
| BAM SSMaxModel size=120M, Training context length=512, Evaluation context length=up to 5122025.05 | 18.6897 | |
| BAMModel size=120M, Training context length=512, Evaluation context length=up to 5122025.05 | 18.9281 | |
| SinusoidalModel size=120M, Training context length=512, Evaluation context length=up to 5122025.05 | 18.931 | |
| Sinusoidal SSMaxModel size=120M, Training context length=512, Evaluation context length=up to 5122025.05 | 19.1467 | |
| Offset-NPO+KLBackbone=Mistral-7B-instruct2024.06 | 19.43 | |
| NoPEModel size=120M, Training context length=512, Evaluation context length=up to 5122025.05 | 19.8609 | |
| RoPE SSMaxModel size=120M, Training context length=512, Evaluation context length=up to 5122025.05 | 20.1854 | |
| NoPE SSMaxModel size=120M, Training context length=512, Evaluation context length=up to 5122025.05 | 20.3899 | |
| Offset-DPO+KLBackbone=Mistral-7B-instruct2024.06 | 23.56 | |
| NPOBackbone=Mistral-7B-instruct2024.06 | 35.71 | |
| DPOBackbone=Mistral-7B-instruct2024.06 | 42.14 | |
| GABackbone=Mistral-7B-instruct2024.06 | 48.13 | |
| Offset-GA+KLBackbone=Mistral-7B-instruct2024.06 | 58.54 |