Question Answering on WMDP Cyber QA
44.3Default AccuracyBase model
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Base modelRefinement=Original, Attack forget-set source=WMDP-Cyber2025.01 | 44.3 | — | — | — | — | |
| Base modelRefinement=Original, Attack forget-set source=WMDP-Biology2025.01 | 44.3 | — | — | — | — | |
| RMURefinement=w/ RNA, Attack forget-set source=WMDP-Cyber2025.01 | 29.1 | 27.7 | 38.3 | 28.5 | 40.5 | |
| RMURefinement=w/ RNA, Attack forget-set source=WMDP-Biology2025.01 | 29.1 | — | 39.5 | 28.6 | 33.4 | |
| RSVRefinement=Original, Attack forget-set source=WMDP-Cyber2025.01 | 28.9 | 27.9 | 41 | 31.7 | 39.8 | |
| RSVRefinement=Original, Attack forget-set source=WMDP-Biology2025.01 | 28.9 | — | 34.5 | 28.8 | 30.6 | |
| Adaptive RMURefinement=w/ RNA, Attack forget-set source=WMDP-Cyber2025.01 | 28.7 | 27.1 | 38.1 | 29.4 | 39.4 | |
| Adaptive RMURefinement=w/ RNA, Attack forget-set source=WMDP-Biology2025.01 | 28.7 | — | 38.6 | 28.4 | 34.2 | |
| RMURefinement=Original, Attack forget-set source=WMDP-Cyber2025.01 | 28.6 | 28.6 | 39.8 | 30.4 | 40.6 | |
| RMURefinement=Original, Attack forget-set source=WMDP-Biology2025.01 | 28.6 | — | 39.7 | 29.9 | 35.5 | |
| RSVRefinement=w/ RNA, Attack forget-set source=WMDP-Cyber2025.01 | 28.2 | 27.5 | 40 | 31.8 | 39.5 | |
| RSVRefinement=w/ RNA, Attack forget-set source=WMDP-Biology2025.01 | 28.2 | — | 35.9 | 29.3 | 31.7 | |
| Adaptive RMURefinement=Original, Attack forget-set source=WMDP-Cyber2025.01 | 27.9 | 28.4 | 39.6 | 32.6 | 41.3 | |
| Adaptive RMURefinement=Original, Attack forget-set source=WMDP-Biology2025.01 | 27.9 | — | 41.4 | 32.5 | 35.4 | |
| SimNPO+KLRefinement=Original, Attack forget-set source=WMDP-Cyber2025.01 | 27.1 | 27.4 | 42.1 | 26.2 | 39.9 | |
| SimNPO+KLRefinement=Original, Attack forget-set source=WMDP-Biology2025.01 | 27.1 | — | 43 | 26.3 | 37.4 | |
| NPO+MSERefinement=w/ RNA, Attack forget-set source=WMDP-Cyber2025.01 | 27 | 27 | 42.9 | 27.6 | 35.7 | |
| DPO+KLRefinement=w/ RNA, Attack forget-set source=WMDP-Cyber2025.01 | 27 | 26.6 | 40.3 | 32.3 | 38.3 | |
| NPO+MSERefinement=w/ RNA, Attack forget-set source=WMDP-Biology2025.01 | 27 | — | 41.2 | 27 | 26.1 | |
| DPO+KLRefinement=w/ RNA, Attack forget-set source=WMDP-Biology2025.01 | 27 | — | 37.3 | 26.9 | 37.7 | |
| SimNPO+MSERefinement=Original, Attack forget-set source=WMDP-Cyber2025.01 | 26.7 | 26.8 | 41.3 | 27.3 | 37 | |
| SimNPO+MSERefinement=Original, Attack forget-set source=WMDP-Biology2025.01 | 26.7 | — | 41.9 | 26.6 | 27.5 | |
| SimNPO+MSERefinement=w/ RNA, Attack forget-set source=WMDP-Cyber2025.01 | 26 | 27.4 | 42.8 | 27.5 | 37.3 | |
| SimNPO+MSERefinement=w/ RNA, Attack forget-set source=WMDP-Biology2025.01 | 26 | — | 41.8 | 26.1 | 36.8 | |
| NPO+KLRefinement=w/ RNA, Attack forget-set source=WMDP-Cyber2025.01 | 25.8 | 27.1 | 43.3 | 28.1 | 39.7 | |
| NPO+KLRefinement=w/ RNA, Attack forget-set source=WMDP-Biology2025.01 | 25.8 | — | 42.5 | 27.2 | 39.9 | |
| SimNPO+KLRefinement=w/ RNA, Attack forget-set source=WMDP-Cyber2025.01 | 25.7 | 27.5 | 42.3 | 26 | 39.8 | |
| SimNPO+KLRefinement=w/ RNA, Attack forget-set source=WMDP-Biology2025.01 | 25.7 | — | 42.4 | 28 | 41.4 | |
| NPO+KLRefinement=Original, Attack forget-set source=WMDP-Cyber2025.01 | 25.6 | 27.5 | 42.8 | 29.8 | 40.6 | |
| NPO+KLRefinement=Original, Attack forget-set source=WMDP-Biology2025.01 | 25.6 | — | 42.5 | 27.2 | 40.5 | |
| NPO+MSERefinement=Original, Attack forget-set source=WMDP-Cyber2025.01 | 25.4 | 27.5 | 42.5 | 25.3 | 36.3 | |
| NPO+MSERefinement=Original, Attack forget-set source=WMDP-Biology2025.01 | 25.4 | — | 41.3 | 27 | 29.2 | |
| DPO+MSERefinement=w/ RNA, Attack forget-set source=WMDP-Cyber2025.01 | 25.2 | 28 | 36.4 | 27.4 | 36.1 | |
| DPO+MSERefinement=w/ RNA, Attack forget-set source=WMDP-Biology2025.01 | 25.2 | — | 25.3 | 27 | 26.2 | |
| DPO+KLRefinement=Original, Attack forget-set source=WMDP-Cyber2025.01 | 25.1 | 27.2 | 40.5 | 28.9 | 35.3 | |
| DPO+KLRefinement=Original, Attack forget-set source=WMDP-Biology2025.01 | 25.1 | — | 38 | 28.4 | 35.3 | |
| DPO+MSERefinement=Original, Attack forget-set source=WMDP-Cyber2025.01 | 23.9 | 26.6 | 40.3 | 26 | 37.4 | |
| DPO+MSERefinement=Original, Attack forget-set source=WMDP-Biology2025.01 | 23.9 | — | 36.9 | 26.4 | 32.4 |