Language Modeling on Alpaca Cleaned (lower-level)
0.835Evaluation LossOnline (R = 10%, G = 5)
Evaluation Results
| Method | Links | |
|---|---|---|
| Online (R = 10%, G = 5)Base LLM=LLAMA-8B, Sample ratio (R)=10%, Generations (G)=52025.11 | 0.835 | |
| Online (R = 10%, G = 1)Base LLM=LLAMA-8B, Sample ratio (R)=10%, Generations (G)=12025.11 | 0.857 | |
| Online (R = 5%, G = 1)Base LLM=LLAMA-8B, Sample ratio (R)=5%, Generations (G)=12025.11 | 0.887 | |
| LESSBase LLM=LLAMA-8B2025.11 | 0.892 | |
| Offline BDSBase LLM=LLAMA-8B2025.11 | 0.893 | |
| DSIRBase LLM=LLAMA-8B2025.11 | 0.896 | |
| Random selectionBase LLM=LLAMA-8B2025.11 | 0.901 | |
| Direct mixing (ρ = 1)Base LLM=LLAMA-8B2025.11 | 0.908 | |
| Direct mixing (ρ = 0.5)Base LLM=LLAMA-8B2025.11 | 0.908 | |
| Online (R = 10%, G = 5)Base LLM=PYTHIA-1B, Sample ratio (R)=10%, Generations (G)=52025.11 | 1.37 | |
| Online (R = 10%, G = 1)Base LLM=PYTHIA-1B, Sample ratio (R)=10%, Generations (G)=12025.11 | 1.41 | |
| Online (R = 5%, G = 1)Base LLM=PYTHIA-1B, Sample ratio (R)=5%, Generations (G)=12025.11 | 1.46 | |
| LESSBase LLM=PYTHIA-1B2025.11 | 1.49 | |
| Offline BDSBase LLM=PYTHIA-1B2025.11 | 1.54 | |
| DSIRBase LLM=PYTHIA-1B2025.11 | 1.57 | |
| Direct mixing (ρ = 0.5)Base LLM=PYTHIA-1B2025.11 | 1.58 | |
| Random selectionBase LLM=PYTHIA-1B2025.11 | 1.61 | |
| Direct mixing (ρ = 1)Base LLM=PYTHIA-1B2025.11 | 1.62 |