# TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies

Source: [TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies](<https://www.sota2.com/research/paper/692d2180d6b0dd3342931d04>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-12-02T05:06:24.125Z

Authors: Guang Liang, Jie Shao, Ningyuan Tang, Xinyao Liu, Jianxin Wu

Published: 2025-11-28T14:33:21.000Z

## Abstract

Native FP8 support in modern hardware is essential for training large Transformers, but is severely hindered by extreme activation outliers. Existing solutions either rely on complex mixed-precision engineering or invasive architectural modifications. This paper fundamentally challenges the conventional wisdom that outliers are data-driven. We demonstrate that extreme outliers are a data-independent, mechanically-produced artifact of training, originating from specific structural properties of the weight matrices (i.e., colinearity). Based on this insight, we propose TWEO (Transformers Without Extreme Outliers), a novel, non-invasive loss function. TWEO effectively prevents extreme outliers via a very simple loss term, which reduces outliers from 10000+ to less than 20. TWEO then enables full-model FP8 pre-training with neither engineering tricks nor architectural changes for both LLM and ViT. When standard FP8 training catastrophically collapses, TWEO achieves performance comparable to the BF16 baseline while delivering a 36% increase in training throughput. Also, TWEO enables a new quantization paradigm. Hardware-friendly W8A8 per-tensor static quantization of LLMs, previously considered completely unusable due to outliers, achieves SOTA performance for the first time on TWEO-trained models.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet (val) (Top-1, Peak, Final)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-val-top-1-peak-final.md>): Image Classification; ImageNet (val); Top-1 Accuracy: 83.4.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2511.23225>)

[PDF](<https://arxiv.org/pdf/2511.23225.pdf>)