Scientific Reasoning on ARCc
53.67AccuracyMixLLM
Evaluation results
| Method | Links | |
|---|---|---|
| MixLLMModel=Llama-3.1-8B, Precision=W4.4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 53.67 | |
| float16Model=Llama-3.1-8B, Precision=float16, Evaluation Protocol (Shots)=0-shot2024.12 | 53.41 | |
| SmoothQuantModel=Llama-3.1-8B, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 53.33 | |
| MixLLMModel=Llama-3.1-8B, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 53.24 | |
| MixLLMModel=Average, Precision=W4.4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 52.7 | |
| MixLLMModel=Mistral-7B-v0.3, Precision=W4.4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 52.47 | |
| float16Model=Mistral-7B-v0.3, Precision=float16, Evaluation Protocol (Shots)=0-shot2024.12 | 52.3 | |
| QuaRotModel=Qwen2.5-7B, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 52.3 | |
| float16Model=Average, Precision=float16, Evaluation Protocol (Shots)=0-shot2024.12 | 52.24 | |
| MixLLMModel=Mistral-7B-v0.3, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 52.13 | |
| MixLLMModel=Average, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 52.1 | |
| MixLLMModel=Qwen2.5-7B, Precision=W4.4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 51.96 | |
| SmoothQuantModel=Mistral-7B-v0.3, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 51.88 | |
| QuaRotModel=Mistral-7B-v0.3, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 51.88 | |
| MixLLMModel=Qwen2.5-7B, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 51.88 | |
| MixLLMModel=Mistral-7B-v0.3, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 51.88 | |
| MixLLMModel=Average, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 51.82 | |
| SmoothQuantModel=Average, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 51.74 | |
| MixLLMModel=Llama-3.1-8B, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 51.71 | |
| QuaRotModel=Average, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 51.39 | |
| QServeModel=Llama-3.1-8B, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 51.28 | |
| QServeModel=Mistral-7B-v0.3, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 51.19 | |
| float16Model=Qwen2.5-7B, Precision=float16, Evaluation Protocol (Shots)=0-shot2024.12 | 51.02 | |
| MixLLMModel=Qwen2.5-7B, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 50.94 | |
| QServeModel=Average, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 50.54 | |
| SmoothQuantModel=Qwen2.5-7B, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 50 | |
| QuaRotModel=Llama-3.1-8B, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 50 | |
| QServeModel=Qwen2.5-7B, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 49.15 | |
| QuaRotModel=Mistral-7B-v0.3, Precision=W4A4, Evaluation Protocol (Shots)=0-shot2024.12 | 48.63 | |
| QuaRotModel=Qwen2.5-7B, Precision=W4A4, Evaluation Protocol (Shots)=0-shot2024.12 | 46.33 | |
| QuaRotModel=Average, Precision=W4A4, Evaluation Protocol (Shots)=0-shot2024.12 | 45.99 | |
| QuaRotModel=Llama-3.1-8B, Precision=W4A4, Evaluation Protocol (Shots)=0-shot2024.12 | 43 |