# Natural Language Understanding on GLUE (Efficiency Metrics) Source: [Natural Language Understanding on GLUE (Efficiency Metrics)]() Collection index: [llms.txt]() Data updated: 2026-02-26T06:03:31.749Z Evaluation of model efficiency, specifically measuring speedup and performance drop across the constituent tasks of the GLUE benchmark suite (SST-2, CoLA, MRPC, QNLI, QQP, RTE, MNLI). A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Natural Language Understanding | | Dataset (including any stated split) | GLUE | | Metric | SST-2 Speedup | | Unit | SCALAR | | Direction | Higher is better | | Results loaded | 32 of 32 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [SST-2 Speedup](): Higher is better; unit: SCALAR. - [SST-2 Performance Drop](): Lower is better; unit: SCALAR. - [CoLA Speedup](): Higher is better; unit: SCALAR. - [CoLA Performance Drop](): Lower is better; unit: SCALAR. - [MRPC Speedup](): Higher is better; unit: SCALAR. - [MRPC Performance Drop](): Lower is better; unit: SCALAR. - [QNLI Speedup](): Higher is better; unit: SCALAR. - [QNLI Performance Drop](): Lower is better; unit: SCALAR. - [QQP Speedup](): Higher is better; unit: SCALAR. - [QQP Performance Drop](): Lower is better; unit: SCALAR. - [RTE Speedup](): Higher is better; unit: SCALAR. - [RTE Performance Drop](): Lower is better; unit: SCALAR. - [MNLI Speedup](): Higher is better; unit: SCALAR. - [MNLI Performance Drop](): Lower is better; unit: SCALAR. - [MNLI-mm Speedup](): Higher is better; unit: SCALAR. - [MNLI-mm Performance Drop](): Lower is better; unit: SCALAR. ## Recorded results | Method | SST-2 Speedup | Result date | Report paper | | --- | --- | --- | --- | | DeeBERT-base | 3.06 | Not recorded | 68fa6f741e209131cea45b28 | | A2T | 3.06 | Not recorded | 68fa6f741e209131cea45b28 | | HotFlip | 2.77 | Not recorded | 68fa6f741e209131cea45b28 | | DeeRoBERTa-base | 2.74 | Not recorded | 68fa6f741e209131cea45b28 | | A2T | 2.7 | Not recorded | 68fa6f741e209131cea45b28 | | PWWS | 2.44 | Not recorded | 68fa6f741e209131cea45b28 | | DeeBERT-base | 2.4 | Not recorded | 68fa6f741e209131cea45b28 | | PWWS | 2.37 | Not recorded | 68fa6f741e209131cea45b28 | | A2T | 2.35 | Not recorded | 68fa6f741e209131cea45b28 | | TextFooler | 2.35 | Not recorded | 68fa6f741e209131cea45b28 | | A2T | 2.34 | Not recorded | 68fa6f741e209131cea45b28 | | DeeRoBERTa-base | 2.31 | Not recorded | 68fa6f741e209131cea45b28 | | TextFooler | 2.3 | Not recorded | 68fa6f741e209131cea45b28 | | TextBugger | 2.29 | Not recorded | 68fa6f741e209131cea45b28 | | HotFlip | 2.28 | Not recorded | 68fa6f741e209131cea45b28 | | TextBugger | 2.25 | Not recorded | 68fa6f741e209131cea45b28 | | HotFlip | 2.21 | Not recorded | 68fa6f741e209131cea45b28 | | PWWS | 2.13 | Not recorded | 68fa6f741e209131cea45b28 | | TextFooler | 2.09 | Not recorded | 68fa6f741e209131cea45b28 | | TextBugger | 2.07 | Not recorded | 68fa6f741e209131cea45b28 | | PWWS | 1.94 | Not recorded | 68fa6f741e209131cea45b28 | | TextBugger | 1.83 | Not recorded | 68fa6f741e209131cea45b28 | | TextFooler | 1.83 | Not recorded | 68fa6f741e209131cea45b28 | | SAME-Word | 1.79 | Not recorded | 68fa6f741e209131cea45b28 | | HotFlip | 1.78 | Not recorded | 68fa6f741e209131cea45b28 | | SAME-Word | 1.63 | Not recorded | 68fa6f741e209131cea45b28 | | SAME-Char | 1.61 | Not recorded | 68fa6f741e209131cea45b28 | | SAME-Char | 1.53 | Not recorded | 68fa6f741e209131cea45b28 | | SAME-Char | 1.21 | Not recorded | 68fa6f741e209131cea45b28 | | SAME-Word | 1.19 | Not recorded | 68fa6f741e209131cea45b28 | | SAME-Char | 1.13 | Not recorded | 68fa6f741e209131cea45b28 | | SAME-Word | 1.1 | Not recorded | 68fa6f741e209131cea45b28 | ### DeeBERT-base SST-2 Speedup: 3.06. Result ID: 6990d2e63a6c314849039133. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### A2T SST-2 Speedup: 3.06. Result ID: 6990d2e63a6c31484903913d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### HotFlip SST-2 Speedup: 2.77. Result ID: 6990d2e63a6c314849039145. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### DeeRoBERTa-base SST-2 Speedup: 2.74. Result ID: 6990d2e63a6c314849039143. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### A2T SST-2 Speedup: 2.7. Result ID: 6990d2e63a6c31484903914d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### PWWS SST-2 Speedup: 2.44. Result ID: 6990d2e63a6c314849039137. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### DeeBERT-base SST-2 Speedup: 2.4. Result ID: 6990d2e63a6c314849039132. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. ### PWWS SST-2 Speedup: 2.37. Result ID: 6990d2e63a6c314849039147. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### A2T SST-2 Speedup: 2.35. Result ID: 6990d2e63a6c31484903913c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. ### TextFooler SST-2 Speedup: 2.35. Result ID: 6990d2e63a6c31484903914b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### A2T SST-2 Speedup: 2.34. Result ID: 6990d2e63a6c31484903914c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. ### DeeRoBERTa-base SST-2 Speedup: 2.31. Result ID: 6990d2e63a6c314849039142. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. ### TextFooler SST-2 Speedup: 2.3. Result ID: 6990d2e63a6c31484903913b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### TextBugger SST-2 Speedup: 2.29. Result ID: 6990d2e63a6c314849039149. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### HotFlip SST-2 Speedup: 2.28. Result ID: 6990d2e63a6c314849039144. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. ### TextBugger SST-2 Speedup: 2.25. Result ID: 6990d2e63a6c314849039139. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### HotFlip SST-2 Speedup: 2.21. Result ID: 6990d2e63a6c314849039135. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### PWWS SST-2 Speedup: 2.13. Result ID: 6990d2e63a6c314849039146. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. ### TextFooler SST-2 Speedup: 2.09. Result ID: 6990d2e63a6c31484903914a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. ### TextBugger SST-2 Speedup: 2.07. Result ID: 6990d2e63a6c314849039148. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. ### PWWS SST-2 Speedup: 1.94. Result ID: 6990d2e63a6c314849039136. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. ### TextBugger SST-2 Speedup: 1.83. Result ID: 6990d2e63a6c314849039138. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. ### TextFooler SST-2 Speedup: 1.83. Result ID: 6990d2e63a6c31484903913a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. ### SAME-Word SST-2 Speedup: 1.79. Result ID: 6990d2e63a6c31484903914f. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### HotFlip SST-2 Speedup: 1.78. Result ID: 6990d2e63a6c314849039134. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. ### SAME-Word SST-2 Speedup: 1.63. Result ID: 6990d2e63a6c31484903914e. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. ### SAME-Char SST-2 Speedup: 1.61. Result ID: 6990d2e63a6c314849039151. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### SAME-Char SST-2 Speedup: 1.53. Result ID: 6990d2e63a6c314849039150. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"DeeRoBERTa-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. ### SAME-Char SST-2 Speedup: 1.21. Result ID: 6990d2e63a6c314849039141. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### SAME-Word SST-2 Speedup: 1.19. Result ID: 6990d2e63a6c31484903913f. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 4%"}. This record does not assert independent reproduction. ### SAME-Char SST-2 Speedup: 1.13. Result ID: 6990d2e63a6c314849039140. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. ### SAME-Word SST-2 Speedup: 1.1. Result ID: 6990d2e63a6c31484903913e. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"DeeBERT-base","Performance Drop Threshold":"\< 2%"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.