Gemma
Benchmarks
TaskDatasetSOTA resultTrendResults
Gemma 1B 3
14.14Input Success Rate
9
Gemma 3 270M SAE Features (Value/Output)
21.82Input Interpretability Score
9
Gemma animal-attribute prompts
26.11Happy Score
9
Gemma-3 27B-it
92ASR
9
Gemma-2B-IT
57.8Utility
8
Gemma-2B-IT 5% forget set
92.4ROUGE-L
8
Gemma-2B-IT 5% forget set
47.1ROUGE-L
8
Gemma 27B-it 3
10Attack Success Rate (ASR)
8
Gemma 27B-it 3
30.2ASR (%)
8
Gemma 2
47Faithfulness
8
Gemma-2 MLP SAE features
49.9Score
8
Gemma-2 Residual SAE features
66.9Score
8
Gemma-2 2B Pre-trained (PT) (test)
100TPR (None)
7
Gemma-2 9B Pre-trained
100TPR
7
Gemma-2 2B Pre-trained
100TPR
7
Gemma-2-2B layer 12 residual-stream activations
0.093Delta CE
6
Gemma-2-2B Layer 0 match with Layer 25
1.83LLM Eval
6
Gemma-2-2B Layer 12 match with Layer 25
1.83LLM Evaluation Score
6
Gemma 9b-it 2
71SRF
6
Gemma-2-2B concept families (across 26 layers)
0.871Correlation (rho) Alin vs delta P
6
Gemma-3-4b-it (test)
5.07Mean Log-Likelihood Difference
6
Gemma (test)
13.26Throughput (tokens/s)
6
Gemma 1B Chat
0.001vNMSE
6
Gemma3 12B-it
0.13TTFT (s)
6
google gemma-3-270m Permute and Flip S1
93.2mIOU
5