C4
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
C4 sampled (test)
6.97Perplexity
8
C4 Llama-3.1-8B
100TPR
8
C4
2.65PPL
7
C4 Vocabulary Size 200,000
0.771AUC
7
C4 Vocabulary Size: 170,000
0.761AUC
7
C4 Vocabulary Size 140,000
0.736AUC
7
C4 Vocabulary Size: 110,000
71.8AUC
7
C4 Vocabulary Size: 80,000
0.693AUC
7
C4 Task-aligned filler (N = 100)
10TPR
6
C4 Toy Problem
96.45Accuracy
6
C4 context length 2048 (test)
8.84Perplexity
6
C4
7.26Perplexity (LLaMA-2 7B/8B)
6
C4 DIPPER attack (test)
1,270TPR @ FPR=0.1%
6
C4 GPT Rephrase attack (test)
16.4TPR (at FPR=0.1%)
6
C4 GPT Back Translation attack (test)
58.9TPR (FPR=0.1%)
6
C4 Random Token Replacement attack (test)
87.5TPR (FPR=0.1%)
6
C4 Synthetic Eval Dataset
0.959BLEU
6
C4 subset
1.927PPL
6
C4
94.9B.Acc (REP 0.05)
5
C4 (test)
100TPR (Substitution 10%)
5
C4 10% insertion attack
100TPR
5
C4 10% deletion attack
100True Positive Rate (TPR)
5
C4 10% substitution attack
100TPR
5
C4 clean setting T = 500
100TPR
5
C4 Regime BM (val)
3.887Final Validation Loss
5