Commonsense Reasoning and Language Modeling on ARC-E, ARC-C, BoolQ, OBQA, HellaSwag, TruthfulQA, MMLU (test)
92.4ARC-E AccuracyREFT
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| REFTBase Model=LLaMA3-8B2024.10 | 92.4 | 81.6 | 75.1 | 87.5 | 96.3 | 53.2 | 55.3 | 77.3 | |
| DORA + MODEKBase Model=LLaMA3-8B2024.10 | 90.8 | 80.8 | 74.9 | 86.2 | 95.8 | 53.1 | 55.2 | 76.7 | |
| DORABase Model=LLaMA3-8B2024.10 | 90.5 | 80.4 | 74.6 | 85.8 | 95.5 | 53 | 55 | 76.4 | |
| REFTBase Model=LLaMA2-7B2024.10 | 87.6 | 61.3 | 72.2 | 83.3 | 93.2 | 49.1 | 37 | 69.1 | |
| DORA + MODEKBase Model=LLaMA2-7B2024.10 | 86 | 58.1 | 73.2 | 83.8 | 88.4 | 49.1 | 37 | 67.9 | |
| DORABase Model=LLaMA2-7B2024.10 | 85.7 | 57.3 | 72.9 | 83.2 | 88 | 49 | 37 | 67.6 | |
| LORAALL + MODEKBase Model=LLaMA3-8B2024.10 | 84.8 | 72 | 71.3 | 80 | 92.2 | 52.3 | 54.2 | 72.4 | |
| LORA¬K + MODEKBase Model=LLaMA3-8B, Additional Trainable Parameters (%)=+0.04%2024.10 | 84.6 | 71.5 | 71 | 79.5 | 91.9 | 52.2 | 54 | 72.1 | |
| LORAALLBase Model=LLaMA3-8B2024.10 | 84.2 | 71.2 | 70.8 | 79 | 91.7 | 52.5 | 54 | 71.9 | |
| LORA¬KBase Model=LLaMA3-8B2024.10 | 83.5 | 70.4 | 70.1 | 78.2 | 90.9 | 51.9 | 53.3 | 71.2 | |
| LORA¬K + MODEKBase Model=LLaMA2-7B, Additional Trainable Parameters (%)=+0.04%2024.10 | 82.9 | 53.4 | 71.5 | 82.1 | 83.5 | 48.9 | 36.9 | 65.6 | |
| LORAALL + MODEKBase Model=LLaMA2-7B2024.10 | 82.2 | 56.4 | 71.4 | 83.4 | 84 | 49.3 | 37.1 | 66.2 | |
| LORAALLBase Model=LLaMA2-7B2024.10 | 81.8 | 53.8 | 70.9 | 82 | 82.5 | 49.6 | 37.3 | 65.4 | |
| LORA¬KBase Model=LLaMA2-7B2024.10 | 75.9 | 48 | 70.5 | 80.4 | 79.9 | 46.5 | 35.8 | 62.4 |