Language Modeling and Reasoning Evaluation on Open LLM Leaderboard
82.8ARCDPO
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| DPOAlignment Algorithm=DPO, Training Data Type=Paired, Base Model=Merlinite-7B, Evaluation Protocol=Zero-shot2024.06 | 82.8 | 50.6 | 74.3 | 52 | 65.8 | 63.1 | — | |
| KTOAlignment Algorithm=KTO, Training Data Type=Unpaired, Base Model=Merlinite-7B, Evaluation Protocol=Zero-shot2024.06 | 82.7 | 48.7 | 74.9 | 53.9 | 65.4 | 63 | — | |
| AOT pairedAlignment Algorithm=AOT, Training Data Type=Paired, Base Model=Merlinite-7B, Evaluation Protocol=Zero-shot2024.06 | 82.5 | 50.8 | 74.4 | 53.1 | 66.1 | 62.9 | — | |
| AOT unpairedAlignment Algorithm=AOT, Training Data Type=Unpaired, Base Model=Merlinite-7B, Evaluation Protocol=Zero-shot2024.06 | 82.5 | 51.1 | 74.4 | 51.8 | 66.2 | 62.8 | — | |
| IPOAlignment Algorithm=IPO, Training Data Type=Paired, Base Model=Merlinite-7B, Evaluation Protocol=Zero-shot2024.06 | 82.4 | 46.5 | 74 | 52.3 | 65.1 | 63 | — | |
| Merlinite-7BAlignment Algorithm=None (SFT), Training Data Type=None, Base Model=Merlinite-7B, Evaluation Protocol=Zero-shot2024.06 | 81.6 | 42 | 73.9 | 45.2 | 63.2 | 62.6 | — | |
| FT# Param=6739M2024.02 | 73.34 | 47.07 | 74.03 | 16.22 | 79.95 | 46.55 | 56.19 | |
| LoRA# Param=8.39M2024.02 | 72.07 | 44.84 | 75.85 | 15.01 | 78.6 | 44.42 | 55.13 | |
| RED# Param=0.26M2024.02 | 72.04 | 47.59 | 72.85 | 11.9 | 77.86 | 42.27 | 54.09 | |
| FP16Model=LLaMA2-13B, Precision=16-bit, Group Size=g642024.04 | 59.4 | 37.4 | 76.6 | 22.5 | 82.1 | 55.7 | 55.6 | |
| FP16Model=LLaMA2-13B, Precision=16-bit, Group Size=g1282024.04 | 59.4 | 37.4 | 76.6 | 22.5 | 82.1 | 55.7 | 55.6 | |
| FP16Backbone=LLaMA2-13B, Quantization=None, Group Size=None2024.04 | 59.4 | 37.4 | 76.6 | 22.5 | 82.1 | 55.7 | 55.6 | |
| CherryQBackbone=LLaMA2-13B, Quantization=4-bit, Group Size=g1282024.04 | 58.6 | 38.8 | 77 | 21 | 82 | 54.6 | 55.3 | |
| QATBackbone=LLaMA2-13B, Quantization=4-bit, Group Size=g1282024.04 | 57.9 | 38.9 | 75.7 | 19.6 | 81.9 | 54.2 | 54.7 | |
| GPTQBackbone=LLaMA2-13B, Quantization=4-bit, Group Size=g1282024.04 | 57.4 | 36.1 | 76.8 | 20.4 | 81.5 | 54.6 | 54.5 | |
| CherryQModel=LLaMA2-13B, Precision=3-bit, Group Size=g642024.04 | 57.3 | 38 | 76.2 | 18.4 | 81.1 | 53.5 | 54.1 | |
| CherryQModel=LLaMA2-13B, Precision=3-bit, Group Size=g1282024.04 | 56.7 | 38.9 | 75.4 | 17.8 | 81 | 52.5 | 53.7 | |
| GPTQModel=LLaMA2-13B, Precision=3-bit, Group Size=g642024.04 | 56.5 | 36 | 74.4 | 16.4 | 79.2 | 52.4 | 52.5 | |
| QATModel=LLaMA2-13B, Precision=3-bit, Group Size=g642024.04 | 55.5 | 39 | 75.1 | 16.8 | 80.7 | 52.9 | 53.3 | |
| QATModel=LLaMA2-13B, Precision=3-bit, Group Size=g1282024.04 | 55.3 | 38.8 | 75.5 | 16 | 80.7 | 51.9 | 53 | |
| GPTQModel=LLaMA2-13B, Precision=3-bit, Group Size=g1282024.04 | 54.1 | 34.9 | 75.4 | 15.6 | 79.1 | 50.3 | 51.6 | |
| FP16Model=LLaMA2-7B, Precision=16-bit, Group Size=g642024.04 | 53.2 | 38.8 | 74 | 14.5 | 78.6 | 46.7 | 51 | |
| FP16Model=LLaMA2-7B, Precision=16-bit, Group Size=g1282024.04 | 53.2 | 38.8 | 74 | 14.5 | 78.6 | 46.7 | 51 | |
| FP16Backbone=LLaMA2-7B, Quantization=None, Group Size=None2024.04 | 53.2 | 38.8 | 74 | 14.5 | 78.6 | 46.7 | 51 | |
| QATBackbone=LLaMA2-7B, Quantization=4-bit, Group Size=g1282024.04 | 52 | 39 | 72.2 | 10.6 | 77.5 | 43.7 | 49.2 | |
| GPTQBackbone=LLaMA2-7B, Quantization=4-bit, Group Size=g1282024.04 | 52 | 39.1 | 72.9 | 11.1 | 77.6 | 43.8 | 49.4 | |
| CherryQBackbone=LLaMA2-7B, Quantization=4-bit, Group Size=g1282024.04 | 51.5 | 39.5 | 73.5 | 12.9 | 77.8 | 44.4 | 49.9 | |
| CherryQModel=LLaMA2-7B, Precision=3-bit, Group Size=g642024.04 | 50.6 | 38.6 | 71.8 | 10.4 | 77 | 43.9 | 48.7 | |
| CherryQModel=LLaMA2-7B, Precision=3-bit, Group Size=g1282024.04 | 49.7 | 38.1 | 72.4 | 8.8 | 76.3 | 41.6 | 47.8 | |
| QATModel=LLaMA2-7B, Precision=3-bit, Group Size=g642024.04 | 49.2 | 37.3 | 71.6 | 7.3 | 75.5 | 40.6 | 46.9 | |
| GPTQModel=LLaMA2-7B, Precision=3-bit, Group Size=g642024.04 | 48.6 | 38.8 | 71.7 | 8.1 | 73.9 | 39.4 | 46.8 | |
| GPTQModel=LLaMA2-7B, Precision=3-bit, Group Size=g1282024.04 | 48.6 | 39.1 | 70.8 | 5.4 | 72.9 | 38.2 | 45.8 | |
| QATModel=LLaMA2-7B, Precision=3-bit, Group Size=g1282024.04 | 48.2 | 37.7 | 70.8 | 6.7 | 75.4 | 39 | 46.3 |