Latent Representation Reconstruction on OpenWebText
230.673L0SMIXAE
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| SMIXAEModel=Gemma 2 2B, Layer=12, Width=6,144, Parameters=151,226,6242026.05 | 230.673 | 97 | 75.2 | 98.4 | 0.189 | 90 | |
| SMIXAEModel=Gemma 2 9B, Layer=11, Width=6,144, Parameters=235,113,9842026.05 | 229.361 | 98.8 | 65.5 | 98.5 | 0.231 | 87.6 | |
| SMIXAEModel=Gemma 2 9B, Layer=20, Width=6,144, Parameters=235,113,9842026.05 | 211.989 | 96.7 | 73.5 | 97.8 | 0.202 | 89.4 | |
| GemmaScope 2B 16kModel=Gemma 2 2B, Layer=12, Width=16,384, Parameters=75,532,5442026.05 | 184.55 | 99.9 | 84.1 | 99.4 | 0.121 | 93.8 | |
| GemmaScope 9B 16kModel=Gemma 2 9B, Layer=20, Width=16,384, Parameters=117,476,8642026.05 | 139.525 | 99.9 | 81.9 | 99.1 | 0.138 | 92.8 | |
| GemmaScope 9B 16kModel=Gemma 2 9B, Layer=11, Width=16,384, Parameters=117,476,8642026.05 | 130.27 | 99.9 | 77.4 | 99.5 | 0.15 | 92.2 |