Question Answering on Natural Questions and TriviaQA 5 shot
41KnowledgeBTX
Evaluation results
| Method | Links | |
|---|---|---|
| BTXActive Experts (Top-k)=22024.03 | 41 | |
| BTXActive Experts (Top-k)=1, Sampling Strategy=Sample2024.03 | 40.1 | |
| LLAMA-2Model Size=13B2024.03 | 40 | |
| DenseTraining Context=Data-Matching (DM)2024.03 | 39.6 | |
| LLAMA-2Model Size=7B2024.03 | 37.4 | |
| Sparse upcyclingTraining Context=Data-Matching (DM), Active Experts (Top-k)=22024.03 | 34 | |
| BTMActive Experts (Top-k)=22024.03 | 26.9 | |
| BTMActive Experts (Top-k)=12024.03 | 26.5 | |
| CODELLAMAModel Size=7B2024.03 | 22.2 | |
| LLEMMAModel Size=7B2024.03 | 17.2 |