Numerical Stability Evaluation on Pretrained Models First Forward Pass
1Max Scaled LogitGeometry-aware Scaling
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Geometry-aware ScalingModel=Llama-2-70B, Layers=80, Attention=Grouped Query Attention, GQA ratio=8:12026.02 | 1 | — | |
| Geometry-aware ScalingModel=Llama-2-13B, Layers=402026.02 | 2.5 | — | |
| Geometry-aware ScalingModel=GPT-2 XL, Parameters=1.5B, Layers=482026.02 | 120 | — | |
| Geometry-aware ScalingModel=Mistral-7B, Layers=32, Attention=Grouped Query Attention, GQA ratio=4:12026.02 | 196 | — | |
| Delayed ScalingModel=Llama-2-13B, Layers=402026.02 | 5,600 | — | |
| Delayed ScalingModel=Mistral-7B, Layers=32, Attention=Grouped Query Attention, GQA ratio=4:12026.02 | 7,123 | — | |
| Delayed ScalingModel=GPT-2 XL, Parameters=1.5B, Layers=482026.02 | 8,761 | — | |
| Delayed ScalingModel=Llama-2-70B, Layers=80, Attention=Grouped Query Attention, GQA ratio=8:12026.02 | 9,498 | — |