Deduplication on RealNews 1M documents
289,229Unique Document CountStandard MinHash
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Standard MinHashType=Oracle/Brute-force2025.01 | 289,229 | — | |
| CPU baselineImplementation=SlimPajama, Hardware=64 logical CPU cores2025.01 | 288,683 | 99.8 | |
| GPU baselineImplementation=NVIDIA NeMo Curator, Hardware=4 GPUs (V100)2025.01 | 283,724 | 96.6 | |
| SEDDHardware=4 GPUs (V100)2025.01 | 283,264 | 96.6 |