Gemma
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
Gemma-2-2B
82Clarity
70
Gemma-2-2B
80Clarity
50
Gemma 9B v2 (test)
100TPR
49
Gemma 2 9B
0Attack Success Rate (ASR)
48
Gemma-2-9B
0ASR
42
GEMMA-3-4B-IT
99.1ROUGE-1 Score
40
Gemma 4B IT 3
98.6ROUGE-1
24
Gemma 2 2B Steering Evaluation Set
1.296Granularity
20
Gemma-2-2B activations
320L0 Count
20
Gemma 4B 3
66NR
20
Gemma-7b five finetuned variants
66.2Average ASR
16
Gemma-2B
92.9Keyword Occurrence (%)
15
gemma-7b v1 (pretrained)
6ASR
13
Gemma-2B Over-refusal
1.6Informative Refusal Rate
12
Gemma-3-4B
94.33Win Rate
12
Gemma-7B Unmodified watermarked text quality
12.188Perplexity (PPL)
11
Gemma-7B Post-translation to French human-authored texts
59.2TPR @ 1% FPR
11
Gemma-7B Post-translation to German human-authored texts
53.4TPR @ 1% FPR
11
Gemma-7B Post-paraphrasing human-authored texts
84TPR @ 1% FPR
11
Gemma-7B Unmodified human-authored texts
100TPR @ 1% FPR
11
Gemma 2 2B
1AUC
10
Gemma 3
47.06Accuracy
10
Gemma-3-4B 16K SAE
14.7Input Success Rate
9
Gemma-3-1B 16K SAE
14.71Input Attribution
9
Gemma-3-270M 16K SAE
17.3Input Success Rate (%)
9