Multi-task Language Evaluation on 16-task evaluation suite 1B/25B checkpoints (test)
42.22Average ScoreHERMES + L12 quality top-30%
Evaluation results
| Method | Links | ||
|---|---|---|---|
| HERMES + L12 quality top-30%L1 outer=DoReMi, Stage-2 sampler=L12 quality top-30%, Quality Reader=Corrected FineWeb-Edu2026.07 | 42.22 | 1.628 | |
| HERMES-256^3 + L1 max-entropyL1 outer=Uniform, Stage-2 sampler=L1 max-entropy2026.07 | 41.55 | 1.048 | |
| KMeans-256 + L1 max-entropyL1 outer=Uniform, Stage-2 sampler=L1 max-entropy2026.07 | 41.53 | 1.059 | |
| WebOrganizer-FormatL1 outer=Uniform on Format, Stage-2 sampler=random within Format2026.07 | 41.34 | 0.949 | |
| WebOrganizer-Topic + Topic DROL1 outer=DoReMi on Topic, Stage-2 sampler=random within Topic2026.07 | 40.82 | 0.508 | |
| WebOrganizer-Topic + Topic max-entropyL1 outer=Uniform on Topic, Stage-2 sampler=random within Topic2026.07 | 40.79 | 0.469 | |
| HERMES + global L123 max-entropy coverageL1 outer=DoReMi, Stage-2 sampler=global L123 max-entropy2026.07 | 40.61 | 0.317 | |
| HERMES + L1-local quality top-30%L1 outer=DoReMi, Stage-2 sampler=L1-local quality top-30%, Quality Reader=Corrected FineWeb-Edu2026.07 | 40.45 | 0.114 | |
| WebOrganizer-Format + Format DROL1 outer=DoReMi on Format, Stage-2 sampler=random within Format2026.07 | 40.29 | 0.128 | |
| HERMES-256^3 + L1 max-entropyL1 outer=DoReMi, Stage-2 sampler=L1 max-entropy2026.07 | 40.28 | 0.048 | |
| HERMES cap-32 + L1 max-entropyL1 outer=Uniform, Stage-2 sampler=L1 max-entropy2026.07 | 39.94 | 0.312 | |
| HERMES cap-64 + L1 max-entropyL1 outer=Uniform, Stage-2 sampler=L1 max-entropy2026.07 | 39.92 | 0.362 | |
| HERMES + L123 quality top-30%L1 outer=DoReMi, Stage-2 sampler=L123 quality top-30%, Quality Reader=Corrected FineWeb-Edu2026.07 | 39.88 | 0.294 | |
| HERMES + per-L1 L123 local random coverageL1 outer=DoReMi, Stage-2 sampler=per-L1 L123 local random2026.07 | 39.86 | 0.302 | |
| HERMES + L12 max-entropy coverageL1 outer=DoReMi, Stage-2 sampler=L12 max-entropy2026.07 | 39.69 | 0.458 | |
| HERMES cap-128 + L1 max-entropyL1 outer=Uniform, Stage-2 sampler=L1 max-entropy2026.07 | 39.34 | 0.747 | |
| KMeans-256 + L1 max-entropyL1 outer=DoReMi, Stage-2 sampler=L1 max-entropy2026.07 | 38.89 | 1.147 | |
| HERMES cap-64 + L1 max-entropyL1 outer=DoReMi, Stage-2 sampler=L1 max-entropy2026.07 | 38.87 | 1.163 | |
| HERMES cap-128 + L1 max-entropyL1 outer=DoReMi, Stage-2 sampler=L1 max-entropy2026.07 | 38.86 | 1.181 | |
| HERMES cap-32 + L1 max-entropyL1 outer=DoReMi, Stage-2 sampler=L1 max-entropy2026.07 | 37.98 | 1.885 |