Language Modeling on Proof-pile (test)
2.91Perplexity (2K)HiCI
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| HiCIBase Model=LLaMA-2-13B, Training Context Length=8K2026.03 | 2.91 | 2.69 | 2.52 | — | — | — | — | |
| HiCIBase Model=LLaMA-2-13B, Training Context Length=32K2026.03 | 2.94 | 2.68 | 2.4 | 2.35 | 2.26 | — | — | |
| HiCIBase Model=LLaMA-2-13B, Training Context Length=64K2026.03 | 2.96 | 2.63 | 2.38 | 2.31 | 2.2 | 2.17 | — | |
| HiCIBase Model=LLaMA-2-13B, Training Context Length=16K2026.03 | 2.99 | 2.73 | 2.53 | 2.4 | — | — | — | |
| LongLoRABase Model=LLaMA-2-13B, Training Context Length=16K2026.03 | 3.03 | 2.74 | 2.55 | 2.41 | — | — | — | |
| LongLoRABase Model=LLaMA-2-13B, Training Context Length=32K2026.03 | 3.03 | 2.74 | 2.55 | 2.41 | 2.32 | — | — | |
| LongLoRABase Model=LLaMA-2-13B, Training Context Length=8K2026.03 | 3.04 | 2.77 | 2.6 | — | — | — | — | |
| LongLoRABase Model=LLaMA-2-13B, Training Context Length=64K2026.03 | 3.05 | 2.76 | 2.57 | 2.42 | 2.32 | 2.25 | — | |
| HiCIBase Model=LLaMA-2-7B, Training Context Length=8K2026.03 | 3.07 | 2.82 | 2.65 | — | — | — | — | |
| HiCIBase Model=LLaMA-2-7B, Training Context Length=16K2026.03 | 3.15 | 2.84 | 2.61 | 2.47 | — | — | — | |
| LongLoRABase Model=LLaMA-2-7B, Training Context Length=16K2026.03 | 3.17 | 2.87 | 2.66 | 2.51 | — | — | — | |
| LongLoRABase Model=LLaMA-2-7B, Training Context Length=8K2026.03 | 3.2 | 2.91 | 2.72 | — | — | — | — | |
| HiCIBase Model=LLaMA-2-7B, Training Context Length=32K2026.03 | 3.21 | 2.87 | 2.71 | 2.58 | 2.49 | — | — | |
| HiCIBase Model=LLaMA-2-7B, Training Context Length=100K2026.03 | 3.27 | 2.86 | 2.73 | 2.54 | 2.48 | 2.46 | 2.43 | |
| LongLoRABase Model=LLaMA-2-7B, Training Context Length=32K2026.03 | 3.35 | 3.01 | 2.78 | 2.61 | 2.5 | — | — | |
| LongLoRABase Model=LLaMA-2-7B, Training Context Length=100K2026.03 | 3.36 | 3.01 | 2.78 | 2.6 | 2.58 | 2.57 | 2.52 |