Language Modeling on Context-length sweep
6.0084Full Perplexity (PPL)Qwen2.5-7B
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Qwen2.5-7BModel=Qwen2.5-7B, Seq=4096, Budget (b)=0.52026.06 | 6.0084 | 2.45 | 1.76 | 16.48 | |
| Qwen2.5-7BModel=Qwen2.5-7B, Seq=8192, Budget (b)=0.52026.06 | 6.2145 | 2.9 | 2.54 | 7.24 | |
| Mistral-7BModel=Mistral-7B, Seq=4096, Budget (b)=0.52026.06 | 6.2379 | 0.13 | 0.03 | 13.69 | |
| Mistral-7BModel=Mistral-7B, Seq=8192, Budget (b)=0.52026.06 | 6.4185 | 0.29 | 0.08 | 7.95 | |
| Qwen2.5-7BModel=Qwen2.5-7B, Seq=2048, Budget (b)=0.52026.06 | 7.0415 | 1.97 | 1.39 | 25.85 | |
| Mistral-7BModel=Mistral-7B, Seq=2048, Budget (b)=0.52026.06 | 7.9568 | 0.06 | 0.1 | 22.07 | |
| Gemma-2-9BModel=Gemma-2-9B, Seq=4096, Budget (b)=0.52026.06 | 7.9672 | 0.14 | 0.2 | 9.31 | |
| Gemma-2-9BModel=Gemma-2-9B, Seq=2048, Budget (b)=0.52026.06 | 11.2042 | 0.44 | 0.47 | 35.63 |