Question Answering on SQuADShifts 1.0 (test) benchmark leaderboard · SOTA2 Research

Question Answering on SQuADShifts 1.0 (test)

54.3Average F1multitask QAG

Evaluation Results

Method	Links
multitask QAG2023.05		54.3	36.7	53.6	34.3	54.1	36.4	57.7	41.8	51.6	34.4
end2end QAG2023.05		53.7	37.3	49.1	31.1	56.1	40.1	60.7	45.5	48.8	32.5
Gold QA2023.05		53.3	37.3	45.9	30.4	55.6	38.7	61.4	46.9	50.1	33.4
pipeline QAG2023.05		52	33.9	50.9	31.1	51.7	32.9	55.9	39.5	49.7	32
end2end QAG2023.05		51.8	35.4	44.9	26.9	56.9	40.1	59.9	45.3	45.3	29.2
pipeline QAG2023.05		51.7	34	49	30.2	52.5	33.7	55.3	40	49.7	32.3
pipeline QAG2023.05		51.7	33.6	50.6	31.1	52.8	34	53.8	37.1	49.6	32.4
multitask QAG2023.05		50.8	33.2	49.4	30.5	50.6	32.1	55	39.2	48.4	31.1
QG only2023.05		50.7	34.7	43.3	27.4	54.4	37.1	57.7	43.2	47.3	31.1
pipeline QAG2023.05		50	32.4	48.4	29.8	49.4	31.1	53	36	49.5	32.7
multitask QAG2023.05		49.6	30.9	48.8	28.9	48.1	28.5	52.5	35.1	49.1	31.2
multitask QAG2023.05		49.5	30.9	46.3	26	49.6	30.6	53	35.9	49	31.2
QG only2023.05		49.4	33.9	42.3	26.7	54.3	37.2	59.3	44.8	41.9	27
QG only2023.05		49.4	33.8	43.3	27.4	54	36.7	59.4	44.6	41.1	26.4
QG only2023.05		48.9	33.4	42.7	26.8	53.2	36.2	58.5	43.9	41.5	26.7
QG only2023.05		48.5	33.1	43.8	27.7	50.5	34.5	55.2	41	44.4	29.1
end2end QAG2023.05		48.3	31.7	42.3	24.8	54.7	37.2	55.2	40.1	41.1	24.8
pipeline QAG2023.05		45.4	27.4	41.6	23.8	46.9	27.2	48.9	32.1	44.1	26.6
multitask QAG2023.05		44	25.2	42.1	22.9	44.3	24.1	45.6	27.9	44	25.7
end2end QAG2023.05		34	21.4	29.3	16.5	35.4	23.2	44.6	31.1	26.6	15
end2end QAG2023.05		17.5	10.2	17.1	9.1	18.9	11.3	22.3	14.7	11.6	5.7