Question Answering on CommonsenseQA (CSQA)
91.2AccuracyDeBERTaV3-large + KEAR
Evaluation results
| Method | Links | |
|---|---|---|
| DeBERTaV3-large + KEAR2023.05 | 91.2 | |
| PaLM 2Number of exemplars (k-shot)=7, Model variant=Instruction-tuned, Chain-of-thought prompting (CoT)=true, Self-consistency (SC)=true2023.05 | 90.4 | |
| PaLM + CoT + SCNumber of exemplars (k-shot)=7, Chain-of-thought prompting (CoT)=true, Self-consistency (SC)=true2023.05 | 80.7 | |
| UNIFIEDQAModel Architecture=T52020.05 | 79.1 | |
| T5Model Architecture=T52020.05 | 78.1 | |
| FreeLB-RoBERTaModel Architecture=RoBERTa2020.05 | 72.2 | |
| LORABackbone Model=LLaMA-1 33B, Precision=16-bit2024.02 | 68.3 | |
| L4QModel=LLaMA-1 33B, Precision=4-bit2024.02 | 67.5 | |
| Pre-trainedBackbone Model=LLaMA-1 33B, Precision=16-bit2024.02 | 67.4 | |
| QAT-LORABackbone Model=LLaMA-1 33B, Precision=3&16-bit2024.02 | 67.4 | |
| L4QBackbone Model=LLaMA-1 33B, Precision=3-bit2024.02 | 67.4 | |
| L4QModel=LLaMA-1 33B, Precision=3-bit2024.02 | 67.4 | |
| LORABackbone Model=LLAMA-3 8B, Precision=16-bit2024.02 | 67.2 | |
| L4QModel=LLAMA-3 8B, Precision=4-bit2024.02 | 66.8 | |
| LORABackbone Model=LLaMA-2 13B, Precision=16-bit2024.02 | 66.5 | |
| LORABackbone Model=Mistral-v0.1 7B, Precision=16-bit2024.02 | 66.4 | |
| Pre-trainedBackbone Model=Mistral-v0.1 7B, Precision=16-bit2024.02 | 66.2 | |
| L4QModel=Mistral-v0.1 7B, Precision=4-bit2024.02 | 66.1 | |
| L4QModel=LLaMA-2 13B, Precision=4-bit2024.02 | 65.8 | |
| Pre-trainedBackbone Model=LLAMA-3 8B, Precision=16-bit2024.02 | 65.6 | |
| LORABackbone Model=LLaMA-1 13B, Precision=16-bit2024.02 | 65.2 | |
| GPTQBackbone Model=LLaMA-1 33B, Precision=3-bit2024.02 | 65.1 | |
| L4QBackbone Model=LLaMA-2 13B, Precision=3-bit2024.02 | 65.1 | |
| L4QModel=LLaMA-2 13B, Precision=3-bit2024.02 | 65.1 | |
| Pre-trainedBackbone Model=LLaMA-2 13B, Precision=16-bit2024.02 | 65 | |
| QA-LORABackbone Model=LLaMA-1 33B, Precision=3-bit2024.02 | 64.6 | |
| L4QModel=LLaMA-1 13B, Precision=4-bit2024.02 | 64.5 | |
| QLORABackbone Model=LLaMA-1 33B, Precision=3&16-bit2024.02 | 64.3 | |
| QAT-LORABackbone Model=LLaMA-2 13B, Precision=3&16-bit2024.02 | 64.3 | |
| UnifiedQA-BARTModel Architecture=BART2020.05 | 64 | |
| Pre-trainedBackbone Model=LLaMA-1 13B, Precision=16-bit2024.02 | 63.8 | |
| L4QModel=LLaMA-2 7B, Precision=4-bit2024.02 | 63.6 | |
| L4QBackbone Model=LLAMA-3 8B, Precision=3-bit2024.02 | 63.5 | |
| L4QModel=LLAMA-3 8B, Precision=3-bit2024.02 | 63.5 | |
| LORABackbone Model=LLaMA-1 7B, Precision=16-bit2024.02 | 63.4 | |
| L4QBackbone Model=LLaMA-1 13B, Precision=3-bit2024.02 | 63.4 | |
| L4QModel=LLaMA-1 13B, Precision=3-bit2024.02 | 63.4 | |
| LORABackbone Model=LLaMA-2 7B, Precision=16-bit2024.02 | 63.3 | |
| QAT-LORABackbone Model=LLaMA-1 13B, Precision=3&16-bit2024.02 | 63.2 | |
| QAT-LORABackbone Model=LLAMA-3 8B, Precision=3&16-bit2024.02 | 63.1 | |
| L4QBackbone Model=Mistral-v0.1 7B, Precision=3-bit2024.02 | 63.1 | |
| L4QModel=Mistral-v0.1 7B, Precision=3-bit2024.02 | 63.1 | |
| QLORABackbone Model=Mistral-v0.1 7B, Precision=3&16-bit2024.02 | 63 | |
| L4QModel=LLaMA-1 7B, Precision=4-bit2024.02 | 62.7 | |
| QLORABackbone Model=LLaMA-2 13B, Precision=3&16-bit2024.02 | 62.5 | |
| BART-largeModel Architecture=BART-large2020.05 | 62.5 | |
| OmniQBackbone Model=LLaMA-1 33B, Precision=3-bit2024.02 | 62.3 | |
| QA-LORABackbone Model=Mistral-v0.1 7B, Precision=3-bit2024.02 | 62.3 | |
| Pre-trainedBackbone Model=LLaMA-2 7B, Precision=16-bit2024.02 | 61.9 | |
| GPTQBackbone Model=Mistral-v0.1 7B, Precision=3-bit2024.02 | 61.8 | |
| Pre-trainedBackbone Model=LLaMA-1 7B, Precision=16-bit2024.02 | 61.7 | |
| GPTQBackbone Model=LLaMA-2 13B, Precision=3-bit2024.02 | 61.7 | |
| QA-LORABackbone Model=LLaMA-2 13B, Precision=3-bit2024.02 | 61.7 | |
| QAT-LORABackbone Model=Mistral-v0.1 7B, Precision=3&16-bit2024.02 | 61.6 | |
| OmniQBackbone Model=Mistral-v0.1 7B, Precision=3-bit2024.02 | 61.4 | |
| QLORABackbone Model=LLaMA-1 13B, Precision=3&16-bit2024.02 | 61.3 | |
| L4QBackbone Model=LLaMA-2 7B, Precision=3-bit2024.02 | 61.3 | |
| L4QModel=LLaMA-2 7B, Precision=3-bit2024.02 | 61.3 | |
| L4QBackbone Model=LLaMA-1 7B, Precision=3-bit2024.02 | 61.2 | |
| L4QModel=LLaMA-1 7B, Precision=3-bit2024.02 | 61.2 | |
| QA-LORABackbone Model=LLaMA-1 13B, Precision=3-bit2024.02 | 61.1 | |
| GPTQBackbone Model=LLaMA-1 13B, Precision=3-bit2024.02 | 61 | |
| QAT-LORABackbone Model=LLaMA-1 7B, Precision=3&16-bit2024.02 | 60.7 | |
| OmniQBackbone Model=LLaMA-2 13B, Precision=3-bit2024.02 | 59.9 | |
| QLORABackbone Model=LLaMA-1 7B, Precision=3&16-bit2024.02 | 59.1 | |
| OmniQBackbone Model=LLaMA-1 13B, Precision=3-bit2024.02 | 58.9 | |
| OmniQBackbone Model=LLAMA-3 8B, Precision=3-bit2024.02 | 58.7 | |
| QA-LORABackbone Model=LLaMA-1 7B, Precision=3-bit2024.02 | 58.7 | |
| LoftQBackbone Model=Mistral-v0.1 7B, Precision=3&16-bit2024.02 | 58.5 | |
| OmniQBackbone Model=LLaMA-2 7B, Precision=3-bit2024.02 | 57.9 | |
| GPTQBackbone Model=LLaMA-2 7B, Precision=3-bit2024.02 | 57.6 | |
| QLORABackbone Model=LLaMA-2 7B, Precision=3&16-bit2024.02 | 57.6 | |
| QAT-LORABackbone Model=LLaMA-2 7B, Precision=3&16-bit2024.02 | 57.4 | |
| QA-LORABackbone Model=LLAMA-3 8B, Precision=3-bit2024.02 | 56.6 | |
| OmniQBackbone Model=LLaMA-1 7B, Precision=3-bit2024.02 | 56.5 | |
| QA-LORABackbone Model=LLaMA-2 7B, Precision=3-bit2024.02 | 56.3 | |
| LORABackbone Model=OpenLLAMA 3B, Precision=16-bit2024.02 | 55.9 | |
| LoRAModel=OpenLLAMA 3B, Precision=16-bit2024.02 | 55.9 | |
| QLORABackbone Model=LLAMA-3 8B, Precision=3&16-bit2024.02 | 55.7 | |
| L4QModel=OpenLLAMA 3B, Precision=4-bit2024.02 | 55 | |
| Pre-trainedBackbone Model=OpenLLAMA 3B, Precision=16-bit2024.02 | 54.8 | |
| LoftQBackbone Model=LLaMA-1 33B, Precision=3&16-bit2024.02 | 54.8 | |
| Pre-trainedModel=OpenLLAMA 3B, Precision=16-bit2024.02 | 54.8 | |
| QAT-LORAModel=OpenLLAMA 3B, Precision=4&16-bit2024.02 | 54.6 | |
| QA-LORAModel=OpenLLAMA 3B, Precision=4-bit2024.02 | 54.5 | |
| QLoRAModel=OpenLLAMA 3B, Precision=4&16-bit2024.02 | 54.4 | |
| LoftQModel=OpenLLAMA 3B, Precision=4&16-bit2024.02 | 54.2 | |
| OmniQModel=OpenLLAMA 3B, Precision=4-bit2024.02 | 54.1 | |
| L4QBackbone Model=OpenLLAMA 3B, Precision=3-bit2024.02 | 54 | |
| LoftQBackbone Model=LLaMA-1 13B, Precision=3&16-bit2024.02 | 54 | |
| L4QModel=OpenLLAMA 3B, Precision=3-bit2024.02 | 54 | |
| GPTQBackbone Model=LLAMA-3 8B, Precision=3-bit2024.02 | 53.5 | |
| GPTQBackbone Model=LLaMA-1 7B, Precision=3-bit2024.02 | 53.4 | |
| QAT-LORABackbone Model=OpenLLAMA 3B, Precision=3&16-bit2024.02 | 53.2 | |
| QAT-LORAModel=OpenLLAMA 3B, Precision=3&16-bit2024.02 | 53.2 | |
| GPTQBackbone Model=OpenLLAMA 3B, Precision=3-bit2024.02 | 52.2 | |
| GPTQModel=OpenLLAMA 3B, Precision=3-bit2024.02 | 52.2 | |
| QA-LORABackbone Model=OpenLLAMA 3B, Precision=3-bit2024.02 | 51.5 | |
| QA-LORAModel=OpenLLAMA 3B, Precision=3-bit2024.02 | 51.5 | |
| QLORABackbone Model=OpenLLAMA 3B, Precision=3&16-bit2024.02 | 51 |