Classification on Analysis N-shot
88AccuracyCRISTAL
Evaluation results
| Method | Links | ||
|---|---|---|---|
| CRISTALN-shots=602026.06 | 88 | 0.81 | |
| CRISTALN-shots=802026.06 | 86 | 0.79 | |
| CRISTALN-shots=202026.06 | 84 | 0.75 | |
| CRISTALN-shots=102026.06 | 83 | 0.74 | |
| CRISTALN-shots=52026.06 | 79 | 0.67 | |
| GPT-4oN-shots=12026.06 | 66 | 0.49 | |
| GPT-4oN-shots=102026.06 | 58 | 0.41 | |
| GPT-4oN-shots=202026.06 | 57 | 0.39 | |
| CRISTALN-shots=12026.06 | 55 | 0.29 | |
| Deepseek-R1 70BN-shots=802026.06 | 43 | 0.25 | |
| Mistral Nemo 3N-shots=52026.06 | 41 | 0.02 | |
| Mistral Nemo 3N-shots=202026.06 | 41 | 0.02 | |
| Qwen2.5-Coder 32BN-shots=802026.06 | 41 | 0.03 | |
| Qwen2.5-Coder 32BN-shots=12026.06 | 39 | -0.01 | |
| Mistral Nemo 3N-shots=802026.06 | 39 | 0 | |
| Mistral Nemo 3N-shots=602026.06 | 38 | -0.02 | |
| Mistral Nemo 3N-shots=12026.06 | 37 | -0.06 | |
| Mistral Nemo 3N-shots=102026.06 | 37 | -0.04 | |
| Llama 3.1 8BN-shots=52026.06 | 36 | 0.01 | |
| Llama 3.1 8BN-shots=12026.06 | 35 | 0 | |
| Deepseek-R1 14BN-shots=102026.06 | 35 | 0.05 | |
| Qwen2.5-Coder 32BN-shots=102026.06 | 35 | -0.05 | |
| Llama 3.1 8BN-shots=202026.06 | 35 | -0.01 | |
| Llama 3.1 8BN-shots=602026.06 | 35 | 0 | |
| Llama 3.1 8BN-shots=102026.06 | 34 | -0.01 | |
| Deepseek-R1 70BN-shots=602026.06 | 34 | 0.12 | |
| Deepseek-R1 70BN-shots=202026.06 | 33 | 0.11 | |
| Deepseek-R1 14BN-shots=802026.06 | 32 | 0.02 | |
| Deepseek-R1 70BN-shots=102026.06 | 31 | 0.06 | |
| Llama 3 70BN-shots=602026.06 | 31 | 0 | |
| Llama 3.1 8BN-shots=802026.06 | 31 | -0.07 | |
| Llama 3 70BN-shots=12026.06 | 30 | 0.01 | |
| Mistral Small 3N-shots=12026.06 | 30 | 0.03 | |
| Qwen2 72BN-shots=12026.06 | 30 | 0.02 | |
| Llama 3 70BN-shots=802026.06 | 30 | 0.01 | |
| Deepseek-R1 14BN-shots=12026.06 | 29 | -0.05 | |
| Mistral Small 3N-shots=102026.06 | 29 | 0.02 | |
| Qwen2 72BN-shots=102026.06 | 29 | 0.01 | |
| Qwen2 72BN-shots=202026.06 | 29 | -0.01 | |
| Mistral Small 3N-shots=602026.06 | 29 | 0.02 | |
| Deepseek-R1 70BN-shots=12026.06 | 28 | 0.05 | |
| Llama 3 70BN-shots=52026.06 | 28 | -0.03 | |
| Mistral Small 3N-shots=52026.06 | 28 | -0.01 | |
| Mistral Small 3N-shots=202026.06 | 28 | -0.01 | |
| Deepseek-R1 70BN-shots=52026.06 | 27 | 0.01 | |
| Llama 3 70BN-shots=102026.06 | 26 | -0.05 | |
| Mistral Small 3N-shots=802026.06 | 26 | -0.02 | |
| Qwen2 72BN-shots=802026.06 | 26 | -0.07 | |
| Llama 3 70BN-shots=202026.06 | 25 | -0.03 |