Continual Model Refinement for Extractive QA on MRQA streams (val)
97.49EFROnline EWC
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Online EWC2022.05 | 97.49 | 73.38 | 86.09 | 56.17 | 47.34 | 65.75 | 68.55 | 85.74 | 53.67 | 53.28 | 65.31 | |
| MaxLossReplay interval (k)=3, Candidate pool size (c)=2562022.05 | 97.43 | 75.43 | 86.89 | 57.14 | 46.7 | 66.54 | 75 | 84.77 | 55.11 | 51.33 | 66.55 | |
| MaxLossReplay interval (k)=1, Candidate pool size (c)=5122022.05 | 97.41 | 75.57 | 87.09 | 56.8 | 46.45 | 66.48 | 77.54 | 89.65 | 55.88 | 52.81 | 68.97 | |
| Continual Fine-Tuning2022.05 | 97.36 | 72.05 | 83.87 | 55.93 | 45.68 | 64.38 | 66.21 | 77.73 | 53.48 | 48.91 | 61.58 | |
| Online L2Reg.2022.05 | 97.18 | 73.47 | 85.37 | 57.27 | 47.12 | 65.81 | 71.09 | 83.59 | 54.5 | 51.17 | 65.09 | |
| MIRReplay interval (k)=3, Candidate pool size (c)=2562022.05 | 97.08 | 75.92 | 87.13 | 56.91 | 47.22 | 66.79 | 75.78 | 87.5 | 54.53 | 51.8 | 67.4 | |
| Experience ReplayReplay interval (k)=32022.05 | 97.07 | 75.3 | 87.29 | 56.02 | 47.61 | 66.55 | 72.46 | 87.3 | 54.08 | 52.66 | 66.63 | |
| MIRReplay interval (k)=1, Candidate pool size (c)=5122022.05 | 96.96 | 77.86 | 89.41 | 58.13 | 46.4 | 67.95 | 79.69 | 89.45 | 55.5 | 50.08 | 68.68 | |
| Experience ReplayReplay interval (k)=12022.05 | 96.72 | 78.91 | 89.38 | 57.8 | 47.17 | 68.31 | 78.13 | 86.52 | 55.33 | 52.73 | 68.18 | |
| MIRReplay interval (k)=1, Candidate pool size (c)=10242022.05 | 96.71 | 77.47 | 87.83 | 57.98 | 46.87 | 67.54 | 78.13 | 87.89 | 55.73 | 50.7 | 68.11 | |
| MaxLossReplay interval (k)=1, Candidate pool size (c)=10242022.05 | 96.63 | 77.61 | 89.82 | 58.13 | 47.1 | 68.17 | 80.66 | 91.02 | 55.88 | 50.78 | 69.59 | |
| MIRReplay interval (k)=1, Candidate pool size (c)=2562022.05 | 96.59 | 77.84 | 89.77 | 58.35 | 47.28 | 68.31 | 79.49 | 90.43 | 55.91 | 51.25 | 69.27 | |
| MaxLossReplay interval (k)=1, Candidate pool size (c)=2562022.05 | 96.54 | 78.16 | 89.86 | 57.78 | 46.63 | 68.11 | 77.54 | 89.26 | 55.47 | 50.94 | 68.3 | |
| MIR(1,256)+OnlineL2RegReplay interval (k)=1, Candidate pool size (c)=2562022.05 | 96.15 | 79.1 | 90.41 | 59.8 | 47.9 | 69.3 | 79.49 | 90.04 | 57.45 | 52.66 | 69.91 | |
| Offline RefiningMode=fo -> fr2022.05 | 95.62 | — | — | — | — | — | 83.78 | 93.75 | 93.81 | 56.17 | 81.88 | |
| Frozen UpstreamMode=ft = fo2022.05 | 0 | 80.27 | 43.69 | 44.95 | 31.25 | 50.04 | 80.27 | 36.13 | 35.44 | 31.25 | 45.77 |