List your company
ResearchFormerly WizWand

Science Question Answering on ARC Easy (0-shot and 32-shot)

87.58AccuracyQwen-1.5 14B
21.64438.76255.8872.998Jan 8, 2024Feb 9, 2024Mar 13, 2024Apr 15, 2024May 18, 2024Jun 20, 2024Jul 23, 2024
Updated 8mo ago

Evaluation results

MethodLinks
Qwen-1.5 14BModel Type=Teacher2024.0787.58——
Qwen-1.5 7BRole=Teacher2024.0785.78——
DDKTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 4B2024.0784.1——
MiniLLMTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 4B2024.0783.92——
Mixtral 8x7BActive Params=13B2024.0183.1——
KDTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 4B2024.0782.19——
Mistral 7BActive Params=7B2024.0180.5——
CPTTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 4B2024.0780.2——
LLaMA 2 70BActive Params=70B2024.0179.9——
LLaMA 1 33BActive Params=33B2024.0179.6——
Qwen-1.5 4BModel Type=Student2024.0779.3——
LLaMA 2 13BActive Params=13B2024.0175.2——
DDKBase Model=Qwen-1.5 1.8B, Teacher=Qwen-1.5 7B2024.0774.43——
MiniLLMBase Model=Qwen-1.5 1.8B, Teacher=Qwen-1.5 7B2024.0773.86——
KDBase Model=Qwen-1.5 1.8B, Teacher=Qwen-1.5 7B2024.0773.74——
LLaMA2-13BModel Size=13B, Shots=0-shot2024.0773.3——
CPTBase Model=Qwen-1.5 1.8B, Teacher=Qwen-1.5 7B2024.0772——
Qwen-1.5 1.8BRole=Student2024.0770.23——
LLaMA 2 7BActive Params=7B2024.0168.7——
TinyLLaMA-1.1B + DDKModel Size=1.1B, Shots=0-shot2024.0736.29——
TinyLLaMA-1.1B + MiniLLMModel Size=1.1B, Shots=0-shot2024.0729.46——
TinyLLaMA-1.1B + KDModel Size=1.1B, Shots=0-shot2024.0729.4——
TinyLLaMA-1.1B + TEDModel Size=1.1B, Shots=0-shot2024.0728.61——
TinyLLaMA-1.1B + CPTModel Size=1.1B, Shots=0-shot2024.0728.06——
TinyLLaMA-1.1B + CPT & DoReMiModel Size=1.1B, Shots=0-shot2024.0727.75——
TinyLLaMA-1.1BModel Size=1.1B, Shots=0-shot2024.0724.18——
OPT 1.3BScale=1.3B2022.12—50.860.2
OPT 175BScale=175B2022.12—6776.8
OPT 30BScale=30B2022.12—63.972.7
OPT-IML 1.3BScale=1.3B2022.12—53.759.9
OPT-IML 175BScale=175B2022.12—70.877.2
OPT-IML 30BScale=30B2022.12—64.972.1