General Language Understanding on ARC-C, CSQA, HellaSwag, LAMBADA, MMLU, OpenBookQA, PIQA, Winogrande (test)
59.5ARC-C AccuracyOriginal
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| OriginalBackbone=Phi-3.5-Mini-Instruct2024.06 | 59.5 | 75.3 | 58.8 | 65.1 | 68.7 | 37.6 | 80 | 74.6 | 65 | |
| OPT-OUTBackbone=Phi-3.5-Mini-Instruct2024.06 | 58.9 | 72.9 | 57.6 | 58.2 | 68 | 36.9 | 80.4 | 74.1 | 63.4 | |
| NPOBackbone=Phi-3.5-Mini-Instruct, Retention Training=true2024.06 | 58.1 | 72.8 | 57.6 | 57.7 | 67.9 | 37.1 | 80 | 74.4 | 63.2 | |
| DPOBackbone=Phi-3.5-Mini-Instruct, Retention Training=true2024.06 | 57.9 | 72.8 | 57.6 | 55.6 | 68 | 37.3 | 80.7 | 75 | 63.1 | |
| IDKBackbone=Phi-3.5-Mini-Instruct, Retention Training=true2024.06 | 57.1 | 74.3 | 57.5 | 55 | 67.7 | 37.7 | 80.2 | 76 | 63.2 | |
| GABackbone=Phi-3.5-Mini-Instruct, Retention Training=true2024.06 | 56.9 | 69.4 | 56.7 | 56.8 | 67.2 | 35.8 | 79.9 | 73.1 | 62 | |
| IDKBackbone=Phi-3.5-Mini-Instruct, Collapsed=true2024.06 | 53.7 | 70.9 | 54.8 | 47.8 | 65.7 | 36.6 | 79.4 | 76.7 | 60.7 | |
| GuardrailBackbone=Phi-3.5-Mini-Instruct2024.06 | 52.2 | 72.9 | 59.2 | 64.1 | 67.7 | 38.3 | 78.1 | 74.5 | 63.4 | |
| OriginalBackbone=Llama-3.1-8B-Instruct2024.06 | 51.8 | 77.1 | 59.2 | 73.2 | 68.1 | 33.8 | 80.2 | 74.1 | 64.7 | |
| NPOBackbone=Llama-3.1-8B-Instruct, Retention Training=true2024.06 | 50.1 | 73.5 | 58.7 | 71.7 | 62.5 | 33.3 | 79.7 | 73 | 62.8 | |
| GuardrailBackbone=Llama-3.1-8B-Instruct2024.06 | 49.8 | 75.5 | 58.9 | 72.3 | 67.2 | 33.2 | 80.1 | 74.2 | 63.9 | |
| OPT-OUTBackbone=Llama-3.1-8B-Instruct2024.06 | 49.8 | 75.3 | 59 | 71.8 | 63.2 | 34.2 | 79.7 | 73.1 | 63.3 | |
| IDKBackbone=Llama-3.1-8B-Instruct, Retention Training=true2024.06 | 49.4 | 73.8 | 58.7 | 69.7 | 63.2 | 34 | 79.8 | 73.4 | 62.8 | |
| DPOBackbone=Llama-3.1-8B-Instruct, Retention Training=true2024.06 | 49.2 | 68.8 | 58.7 | 68.6 | 57.9 | 33.6 | 79.8 | 72.7 | 61.1 | |
| GABackbone=Llama-3.1-8B-Instruct, Retention Training=true2024.06 | 47.4 | 71 | 57.7 | 71.1 | 60.7 | 32.9 | 79.2 | 72.2 | 61.5 | |
| DPOBackbone=Phi-3.5-Mini-Instruct, Collapsed=true2024.06 | 38.9 | 36.1 | 56.7 | 54.3 | 54 | 36.4 | 63.9 | 72.7 | 51.6 | |
| IDKBackbone=Llama-3.1-8B-Instruct, Collapsed=true2024.06 | 38.4 | 67.6 | 53.3 | 48 | 61.8 | 30 | 77.1 | 71.3 | 56 | |
| NPOBackbone=Phi-3.5-Mini-Instruct, Collapsed=true2024.06 | 28.6 | 19.6 | 56.8 | 57 | 26.6 | 35.8 | 59.1 | 69.4 | 44.1 | |
| NPOBackbone=Llama-3.1-8B-Instruct, Collapsed=true2024.06 | 24.7 | 23 | 37.6 | 20.8 | 36.3 | 30.6 | 60.1 | 67.2 | 37.5 | |
| GABackbone=Phi-3.5-Mini-Instruct2024.06 | 24 | 19.8 | 52.1 | 47 | 23.6 | 32.6 | 53.8 | 66 | 39.8 | |
| GABackbone=Llama-3.1-8B-Instruct, Collapsed=true2024.06 | 23.6 | 21.9 | 32.2 | 11.6 | 33.9 | 28 | 58.3 | 60.9 | 33.8 | |
| DPOBackbone=Llama-3.1-8B-Instruct, Collapsed=true2024.06 | 22.9 | 30.5 | 52.8 | 36.4 | 37.9 | 28.3 | 58.4 | 69.5 | 42.1 |