Deduplication on RealNews 0.1M documents
7,158Duplication RatioGPU baseline
Evaluation results
| Method | Links | ||
|---|---|---|---|
| GPU baselineImplementation=NVIDIA NeMo Curator, Hardware=4 GPUs (V100)2025.01 | 7,158 | 95.2 | |
| SEDDHardware=4 GPUs (V100)2025.01 | 7,176 | 95.6 | |
| CPU baselineImplementation=SlimPajama, Hardware=64 logical CPU cores2025.01 | 7,284 | 99.5 | |
| Standard MinHashType=Oracle/Brute-force2025.01 | 7,317 | — |