Natural Language Understanding on GLUE (Efficiency Metrics)

3.06SST-2 SpeedupDeeBERT-base

Evaluation Results

Method
DeeBERT-base2023.05	3.06	0.8	1.25	25.98	1.98	6.86	1.71	5.39	3.19	1.4	1.21	59.93	1.54	18	1.54	18
A2T2023.05	3.06	3.1	1.25	10.16	1.32	18.5	1.63	27	3.35	1.8	1.66	21.66	1.44	26.6	1.43	25.8
HotFlip2023.05	2.77	0.69	1.2	18.31	1.69	1.23	1.65	18.87	7.78	0.7	1.4	0	1.32	3.9	1.37	0.4
DeeRoBERTa-base2023.05	2.74	0	1.3	3.26	1.86	0.49	1.61	2.3	2.35	1.4	1.42	0	1.38	0.72	1.36	1.5
A2T2023.05	2.7	0.34	1.25	10.16	1.89	2.21	1.63	1.8	2.52	3.4	1.37	0.4	1.35	1.9	1.31	4.2
PWWS2023.05	2.44	4.01	1.22	10.93	1.4	5.85	1.71	20.32	3.35	1.5	1.71	16.61	1.34	37.1	1.36	37.5
DeeBERT-base2023.05	2.4	2.87	1.36	8.72	1.51	20.34	1.69	1.23	2.68	3.6	1.19	61.01	1.42	25.6	1.42	25.6
PWWS2023.05	2.37	0.34	1.22	10.93	1.71	5.39	1.56	2.9	2.51	1.3	1.38	0.72	1.3	4.7	1.37	2.3
A2T2023.05	2.35	6.77	1.24	13.23	1.27	51.96	1.58	26.72	2.7	5.2	1.18	65.34	1.3	43.8	1.3	39.3
TextFooler2023.05	2.35	0.92	1.22	11.22	1.63	8.09	1.51	3.7	2.44	3.3	1.34	0.7	1.3	4.7	1.33	3.2
A2T2023.05	2.34	0.8	1.24	13.23	1.52	6.62	1.46	5.7	2.15	6.5	1.29	10.47	1.33	3.4	1.32	5.2
DeeRoBERTa-base2023.05	2.31	0.34	1.29	4.6	1.63	0.74	1.61	0.4	2.09	2.3	1.32	3.97	1.4	0	1.35	1.6
TextFooler2023.05	2.3	5.62	1.22	11.22	1.31	12.94	1.64	29.6	3.19	2.1	1.56	28.52	1.36	37.5	1.32	38.9
TextBugger2023.05	2.29	0.57	1.2	15.92	1.61	8.09	1.5	4	2.78	1.6	1.41	0	1.19	34.2	1.38	1
HotFlip2023.05	2.28	1.49	1.19	22.15	1.58	2.21	1.58	26.72	6.48	1.3	1.29	9.39	1.31	5.2	1.34	0.7
TextBugger2023.05	2.25	5.85	1.2	15.92	1.31	15.72	1.63	21.32	3.24	2.4	1.54	31.05	1.33	39.1	1.32	41.8
HotFlip2023.05	2.21	7.11	1.2	18.31	1.31	12.18	1.63	8.09	3.13	2.6	1.71	14.08	1.42	25.6	1.42	25.6
PWWS2023.05	2.13	0.92	1.21	14.29	1.5	8.33	1.48	1.3	2.15	2.8	1.28	12.27	1.31	6.3	1.37	1.3
TextFooler2023.05	2.09	1.26	1.21	12.08	1.59	3.43	1.4	7.5	2.08	5	1.26	17.69	1.31	6.3	1.32	2.8
TextBugger2023.05	2.07	1.95	1.19	19.27	1.52	6.37	1.45	7.9	2.32	3.8	1.26	17.69	1.33	3.4	1.36	2
PWWS2023.05	1.94	10.32	1.21	14.29	1.31	12.18	1.46	39.4	2.77	3.7	1.18	66.79	1.25	48.5	1.25	48.5
TextBugger2023.05	1.83	14.11	1.19	19.27	1.43	31.62	1.41	44.1	2.66	5.4	1.18	67.15	1.37	34.2	1.33	39.1
TextFooler2023.05	1.83	15.48	1.21	12.08	1.31	43.38	1.41	42.9	2.57	6.3	1.18	67.15	1.24	53.9	1.24	53.9
SAME-Word2023.05	1.79	10.09	1	98.75	1.39	21.57	1.11	82	1.43	47.5	1.28	15.88	1.12	56.1	1.17	42.1
HotFlip2023.05	1.78	17.09	1.19	22.15	1.27	22.72	1.61	8.09	2.58	7	1.18	67.15	1.34	37.1	1.34	37.1
SAME-Word2023.05	1.63	16.74	1.01	95.11	1.52	14.95	1.12	81.9	1.33	52.9	1.14	49.82	1.17	39.8	1.21	28.8
SAME-Char2023.05	1.61	12.5	1	98.18	1.21	49.02	1.24	35	1.47	47.3	1.22	26.35	1.33	2.9	1.17	42.1
SAME-Char2023.05	1.53	18.12	1.01	94.53	1.26	43.87	1.2	39.2	1.34	55.5	1.07	73.29	1.13	52.3	1.13	52.3
SAME-Char2023.05	1.21	59.17	1.01	95.69	1.08	81.37	1.11	82	1.53	45.5	1.07	87	1.02	94.4	1.02	94.8
SAME-Word2023.05	1.19	62.96	1.02	92.91	1.09	77.94	1.18	74.4	1.36	50.6	1.08	84.48	1.01	96	1.02	95.4
SAME-Char2023.05	1.13	70.87	1.01	96.84	1.02	93.14	1.08	88.4	1.37	55.7	1.02	96.03	1.01	96.9	1.01	95.7
SAME-Word2023.05	1.1	76.72	1.02	94.15	1.02	94.61	1.12	81.9	1.25	63.1	1.02	97.11	1.01	96.5	1.01	97.7