Natural Language Inference on MNLI matched mismatched (dev)
84Accuracy (Matched)Baseline
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Baselinew-bits=32, e-bits=32, Size (MB)=415.4, Size-w/o-e (MB)=324.52019.09 | 84 | 84.4 | |
| Q-BERTw-bits=8, e-bits=8, Size (MB)=103.9, Size-w/o-e (MB)=81.22019.09 | 83.91 | 83.83 | |
| Q-BERTw-bits=4, e-bits=8, Size (MB)=63.4, Size-w/o-e (MB)=40.62019.09 | 83.89 | 84.17 | |
| Q-BERT_MPw-bits=2/4 MP, e-bits=8, Size (MB)=53.2, Size-w/o-e (MB)=30.52019.09 | 83.51 | 83.55 | |
| Q-BERTw-bits=3, e-bits=8, Size (MB)=53.2, Size-w/o-e (MB)=30.52019.09 | 83.41 | 83.83 | |
| Q-BERT_MPw-bits=2/3 MP, e-bits=8, Size (MB)=46.1, Size-w/o-e (MB)=23.42019.09 | 81.75 | 82.29 | |
| DirectQw-bits=4, e-bits=8, Size (MB)=63.4, Size-w/o-e (MB)=40.62019.09 | 76.69 | 77 | |
| Q-BERTw-bits=2, e-bits=8, Size (MB)=43.1, Size-w/o-e (MB)=20.42019.09 | 76.56 | 77.02 | |
| DirectQw-bits=3, e-bits=8, Size (MB)=53.2, Size-w/o-e (MB)=30.52019.09 | 70.27 | 70.89 | |
| DirectQw-bits=2, e-bits=8, Size (MB)=43.1, Size-w/o-e (MB)=20.42019.09 | 53.29 | 53.32 |