Next Token Prediction Perturbation
0.014HS-MSERollout over layers
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Rollout over layersLLM=Pico, Configuration=W-Attn2026.01 | 0.014 | 0.02 | |
| Rollout over layersLLM=Pico, Configuration=Attn2026.01 | 0.03 | 0.019 | |
| Rollout over layersLLM=Pico, Configuration=GlbEnc2026.01 | 0.036 | 0.036 | |
| Rollout over layersLLM=Phi, Configuration=W-AttnResLN2026.01 | 0.053 | 0.026 | |
| Rollout over layersLLM=Phi, Configuration=W-Attn2026.01 | 0.067 | 0.023 | |
| Rollout over layersLLM=Phi, Configuration=Attn2026.01 | 0.079 | 0.028 | |
| Rollout over layersLLM=Pico, Configuration=W-AttnResLN2026.01 | 0.09 | 0.067 | |
| Mean over heads & layersLLM=Pico, Configuration=Attn2026.01 | 0.211 | 0.125 | |
| Rollout over layersLLM=Pythia, Configuration=W-Attn2026.01 | 0.213 | 0.035 | |
| TensorLensLLM=Pico, Configuration=In+Class2026.01 | 0.22 | 0.129 | |
| Mean over heads & layersLLM=Pico, Configuration=W-Attn2026.01 | 0.22 | 0.127 | |
| TensorLensLLM=Pico, Configuration=Norm2026.01 | 0.222 | 0.129 | |
| Mean over heads & layersLLM=Pico, Configuration=W-AttnResLN2026.01 | 0.222 | 0.129 | |
| TensorLensLLM=Pico, Configuration=In+Out2026.01 | 0.223 | 0.128 | |
| Mean over heads & layersLLM=Pico, Configuration=GlbEnc2026.01 | 0.225 | 0.129 | |
| Rollout over layersLLM=Pythia, Configuration=W-AttnResLN2026.01 | 0.233 | 0.037 | |
| Rollout over layersLLM=Pythia, Configuration=Attn2026.01 | 0.306 | 0.044 | |
| Mean over heads & layersLLM=Phi, Configuration=Attn2026.01 | 0.864 | 0.133 | |
| TensorLensLLM=Phi, Configuration=In+Class2026.01 | 0.886 | 0.143 | |
| TensorLensLLM=Phi, Configuration=In+Out2026.01 | 0.887 | 0.143 | |
| TensorLensLLM=Phi, Configuration=Norm2026.01 | 0.892 | 0.141 | |
| Mean over heads & layersLLM=Phi, Configuration=W-Attn2026.01 | 0.905 | 0.141 | |
| Mean over heads & layersLLM=Phi, Configuration=W-AttnResLN2026.01 | 0.934 | 0.144 | |
| Mean over heads & layersLLM=Pythia, Configuration=Attn2026.01 | 3.483 | 0.141 | |
| TensorLensLLM=Pythia, Configuration=In+Out2026.01 | 3.603 | 0.146 | |
| Mean over heads & layersLLM=Pythia, Configuration=W-Attn2026.01 | 3.65 | 0.147 | |
| TensorLensLLM=Pythia, Configuration=In+Class2026.01 | 3.683 | 0.147 | |
| TensorLensLLM=Pythia, Configuration=Norm2026.01 | 3.708 | 0.147 | |
| Mean over heads & layersLLM=Pythia, Configuration=W-AttnResLN2026.01 | 3.708 | 0.148 |