Document Deduplication on RedPajama 100GB web content
94SASemHash-LLM
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| SemHash-LLMmode=full2026.07 | 94 | 91 | 89 | 92 | 88 | 91.05 | |
| SemHash-LLMablation=without AW-MinHash2026.07 | 89 | 85 | 82 | 88 | 83 | 85.18 | |
| DedupLM2026.07 | 88 | 83 | 81 | 76 | 71 | 81.2 | |
| SemHash-LLMablation=without MGFN2026.07 | 86 | 82 | 80 | 85 | 79 | 84.56 | |
| NearDup-BERTBackbone=BERT2026.07 | 85 | 79 | 74 | 68 | 55 | 73.45 | |
| SemHash-LLMablation=without SPH2026.07 | 83 | 80 | 79 | 74 | 70 | 79.32 | |
| MinHash-LSH2026.07 | 82 | 78 | 68 | 45 | 22 | 61.35 | |
| SimHash-64Hashing Bits=642026.07 | 71 | 65 | 42 | 31 | 18 | 48.2 |