Multiple-choice Question Answering on CommonsenseQA (CSQA)
66.4AccuracyLlama-2-13b-chat (OTTER)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Llama-2-13b-chat (OTTER)Backbone=Llama-2-13b-chat, Evaluation protocol=Zero-shot, Variant=OTTER2024.04 | 66.4 | 1.8 | |
| Vicuna-13b (OTTER)Backbone=Vicuna-13b, Evaluation protocol=Zero-shot, Variant=OTTER2024.04 | 64.5 | 3.1 | |
| Llama-2-13b-chat (Naive)Backbone=Llama-2-13b-chat, Evaluation protocol=Zero-shot, Variant=Naive2024.04 | 64 | 9.8 | |
| Vicuna-13b (Naive)Backbone=Vicuna-13b, Evaluation protocol=Zero-shot, Variant=Naive2024.04 | 63.4 | 12.9 | |
| Llama-2-7b-chat (OTTER)Backbone=Llama-2-7b-chat, Evaluation protocol=Zero-shot, Variant=OTTER2024.04 | 60.4 | 3.5 | |
| Llama-2-13b (OTTER)Backbone=Llama-2-13b, Evaluation protocol=Zero-shot, Variant=OTTER2024.04 | 58.1 | 2 | |
| Vicuna-7b (OTTER)Backbone=Vicuna-7b, Evaluation protocol=Zero-shot, Variant=OTTER2024.04 | 57.6 | 1 | |
| Llama-2-13b (Naive)Backbone=Llama-2-13b, Evaluation protocol=Zero-shot, Variant=Naive2024.04 | 57 | 10.2 | |
| Vicuna-7b (Naive)Backbone=Vicuna-7b, Evaluation protocol=Zero-shot, Variant=Naive2024.04 | 56.9 | 8.3 | |
| Llama-2-7b-chat (Naive)Backbone=Llama-2-7b-chat, Evaluation protocol=Zero-shot, Variant=Naive2024.04 | 56.5 | 15.2 | |
| Llama-2-7b (OTTER)Backbone=Llama-2-7b, Evaluation protocol=Zero-shot, Variant=OTTER2024.04 | 42.7 | 3.8 | |
| Conventional (OLMo-2)Size=1074M2026.02 | 37.3 | — | |
| ConventionalSize=1074M2026.02 | 36.9 | — | |
| ConventionalSize=906M2026.02 | 36.2 | — | |
| HourglassSize=906M2026.02 | 36.1 | — | |
| HourglassSize=1074M2026.02 | 35.9 | — | |
| HourglassSize=403M2026.02 | 33.3 | — | |
| ConventionalSize=403M2026.02 | 33.2 | — | |
| Llama-2-7b (Naive)Backbone=Llama-2-7b, Evaluation protocol=Zero-shot, Variant=Naive2024.04 | 31.9 | 28.4 | |
| HourglassSize=113M2026.02 | 29.6 | — | |
| ConventionalSize=113M2026.02 | 29.3 | — |