Question Answering on QA Tasks (OpenBookQA, PIQA, ARC-E, ARC-C, SciQ, WebQs) (test)
0.394OpenBookQAOurs (LoRA)
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Ours (LoRA)Base Model=OPT-6.7B, #Trainable Params=5.2M, #Foundation Model Params=52%2024.06 | 0.394 | 0.749 | 0.634 | 0.327 | 0.929 | 0.358 | 0.565 | |
| LoRA (r=64)Base Model=OPT-6.7B, #Trainable Params=33.6M, #Foundation Model Params=100%2024.06 | 0.392 | 0.785 | 0.675 | 0.367 | 0.94 | 0.385 | 0.591 | |
| Ours (LoRA)Base Model=OPT-6.7B, #Trainable Params=2.0M, #Foundation Model Params=52%2024.06 | 0.372 | 0.76 | 0.644 | 0.317 | 0.933 | 0.347 | 0.562 | |
| Offsite-TuningBase Model=OPT-6.7B, #Foundation Model Params=100%2024.06 | 0.338 | 0.777 | 0.668 | 0.339 | 0.919 | 0.239 | 0.547 | |
| LoRA (r=64)Base Model=OPT-1.3B, #Trainable Params=12.6M, #Foundation Model Params=100%2024.06 | 0.336 | 0.747 | 0.595 | 0.295 | 0.92 | 0.298 | 0.532 | |
| Ours (LoRA)Base Model=OPT-1.3B, #Trainable Params=1.5M, #Foundation Model Params=64%2024.06 | 0.332 | 0.741 | 0.59 | 0.284 | 0.927 | 0.286 | 0.527 | |
| Ours (LoRA)Base Model=OPT-1.3B, #Trainable Params=1.9M, #Foundation Model Params=54%2024.06 | 0.332 | 0.726 | 0.576 | 0.275 | 0.918 | 0.282 | 0.518 | |
| Full-FTBase Model=OPT-1.3B, #Trainable Params=1.3B, #Foundation Model Params=100%2024.06 | 0.314 | 0.752 | 0.613 | 0.277 | 0.925 | 0.312 | 0.532 | |
| LLM-Pruner (sp.)Base Model=OPT-1.3B, #Trainable Params=10.6M, #Foundation Model Params=70%2024.06 | 0.304 | 0.729 | 0.559 | 0.276 | 0.887 | 0.265 | 0.503 | |
| LoRA (r=8)Base Model=OPT-1.3B, #Trainable Params=1.6M, #Foundation Model Params=100%2024.06 | 0.296 | 0.746 | 0.599 | 0.291 | 0.93 | 0.287 | 0.525 | |
| Offsite-TuningBase Model=OPT-1.3B, #Foundation Model Params=100%2024.06 | 0.29 | 0.745 | 0.594 | 0.278 | 0.929 | 0.262 | 0.516 | |
| LLM-Pruner (ag.)Base Model=OPT-1.3B, #Trainable Params=10.6M, #Foundation Model Params=70%2024.06 | 0.29 | 0.724 | 0.54 | 0.247 | 0.892 | 0.207 | 0.483 |