Near-duplicate identification on C4 Sampled 50M documents
2,766,436Number of DuplicatesSEDD
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| SEDD2025.01 | 2,766,436 | 2,486,835 | 98.4 | |
| GPU baseline (NeMo Curator)2025.01 | 2,526,120 | — | — |
Loading the SOTA2 catalog…
| Method | Links | |||
|---|---|---|---|---|
| SEDD2025.01 | 2,766,436 | 2,486,835 | 98.4 | |
| GPU baseline (NeMo Curator)2025.01 | 2,526,120 | — | — |