Language Modeling on OpenOrca (upper-level)
0.76Evaluation LossOnline (R = 10%, G = 5)
Evaluation Results
| Method | Links | |
|---|---|---|
| Online (R = 10%, G = 5)Base LLM=LLAMA-8B, Sample ratio (R)=10%, Generations (G)=52025.11 | 0.76 | |
| Online (R = 10%, G = 1)Base LLM=LLAMA-8B, Sample ratio (R)=10%, Generations (G)=12025.11 | 0.78 | |
| Offline BDSBase LLM=LLAMA-8B2025.11 | 0.8 | |
| Online (R = 5%, G = 1)Base LLM=LLAMA-8B, Sample ratio (R)=5%, Generations (G)=12025.11 | 0.8 | |
| Direct mixing (ρ = 0.5)Base LLM=LLAMA-8B2025.11 | 0.84 | |
| LESSBase LLM=LLAMA-8B2025.11 | 0.88 | |
| Direct mixing (ρ = 1)Base LLM=LLAMA-8B2025.11 | 0.92 | |
| DSIRBase LLM=LLAMA-8B2025.11 | 0.92 | |
| Random selectionBase LLM=LLAMA-8B2025.11 | 0.94 | |
| Online (R = 10%, G = 5)Base LLM=PYTHIA-1B, Sample ratio (R)=10%, Generations (G)=52025.11 | 1.32 | |
| Online (R = 10%, G = 1)Base LLM=PYTHIA-1B, Sample ratio (R)=10%, Generations (G)=12025.11 | 1.34 | |
| Online (R = 5%, G = 1)Base LLM=PYTHIA-1B, Sample ratio (R)=5%, Generations (G)=12025.11 | 1.37 | |
| Offline BDSBase LLM=PYTHIA-1B2025.11 | 1.38 | |
| Direct mixing (ρ = 0.5)Base LLM=PYTHIA-1B2025.11 | 1.41 | |
| DSIRBase LLM=PYTHIA-1B2025.11 | 1.52 | |
| LESSBase LLM=PYTHIA-1B2025.11 | 1.54 | |
| Random selectionBase LLM=PYTHIA-1B2025.11 | 1.55 | |
| Direct mixing (ρ = 1)Base LLM=PYTHIA-1B2025.11 | 1.56 |