Knowledge Retention on Llama 8B Pretraining Distribution 3.1
12.3Retain (pp)LoRA
Evaluation results
| Method | Links | |
|---|---|---|
| LoRABackbone=Llama-3.1-8B, Rank (r)=16, Number of seeds=32026.04 | 12.3 | |
| Proj-LoRABackbone=Llama-3.1-8B, Rank (r)=16, Number of seeds=32026.04 | 8.9 | |
| PiSSABackbone=Llama-3.1-8B, Rank (r)=16, Number of seeds=32026.04 | 7.1 | |
| HiP-LoRABackbone=Llama-3.1-8B, Rank (r)=16, k (spectral budget)=32, Number of seeds=32026.04 | 4.9 |