Multiple-choice Question Answering on MMLU (test)
86.4AccuracyGPT-4
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-4number_of_shots=5-shot2023.03 | 86.4 | — | — | — | — | — | — | — | — | — | — | |
| Flan-PaLMnumber_of_shots=5-shot2023.03 | 75.2 | — | — | — | — | — | — | — | — | — | — | |
| U-PaLMnumber_of_shots=5-shot2023.03 | 70.7 | — | — | — | — | — | — | — | — | — | — | |
| GPT-3.5number_of_shots=5-shot2023.03 | 70 | — | — | — | — | — | — | — | — | — | — | |
| PPT (Distribution)Backbone=QWEN2-7B2026.05 | 65.5 | — | — | — | — | — | — | — | — | — | — | |
| BaseBackbone=QWEN2-7B2026.05 | 65.1 | — | — | — | — | — | — | — | — | — | — | |
| PPT (Mean)Backbone=QWEN2-7B2026.05 | 64.9 | — | — | — | — | — | — | — | — | — | — | |
| PPT (Distribution)Backbone=LLAMA3-8B2026.05 | 61.2 | — | — | — | — | — | — | — | — | — | — | |
| BaseBackbone=LLAMA3-8B2026.05 | 60 | — | — | — | — | — | — | — | — | — | — | |
| PPT (Mean)Backbone=LLAMA3-8B2026.05 | 59.5 | — | — | — | — | — | — | — | — | — | — | |
| VerbalizedBackbone=LLAMA3-8B2026.05 | 42.8 | — | — | — | — | — | — | — | — | — | — | |
| BADSBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 26.59 | — | — | — | — | — | — | — | — | — | — | |
| AMOOptimizer=AMO, Model=Llama3.1-760M, Few-shot=02026.05 | 26.24 | — | — | — | — | — | — | — | — | — | — | |
| Random_SelectBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 25.62 | — | — | — | — | — | — | — | — | — | — | |
| CDSBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 25.62 | — | — | — | — | — | — | — | — | — | — | |
| AMOOptimizer=AMO, Model=Llama3.1-1.4B, Few-shot=02026.05 | 25.6 | — | — | — | — | — | — | — | — | — | — | |
| AskLLM-OBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 25.55 | — | — | — | — | — | — | — | — | — | — | |
| Meta_OnlyBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 25.51 | — | — | — | — | — | — | — | — | — | — | |
| Duplicate_MetaBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 25.3 | — | — | — | — | — | — | — | — | — | — | |
| MixingBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 25.16 | — | — | — | — | — | — | — | — | — | — | |
| ClassActBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 24.9 | — | — | — | — | — | — | — | — | — | — | |
| ACUTEvariant=early act2026.06 | — | — | — | — | — | — | 7 | 0.91 | 0.73 | 0.74 | 0.79 | |
| ACUTEvariant=mid act2026.06 | — | — | — | — | — | — | 7 | 0.91 | 0.76 | 0.78 | 0.82 | |
| ACUTEvariant=late act2026.06 | — | — | — | — | — | — | 7 | 0.91 | 0.77 | 0.8 | 0.83 | |
| ACUTEvariant=cosine2026.06 | — | — | — | — | — | — | 9 | 0.9 | 0.75 | 0.78 | 0.81 | |
| ACUTEvariant=pca102026.06 | — | — | — | — | — | — | 8 | 0.91 | 0.76 | 0.78 | 0.82 | |
| ACUTEvariant=pca202026.06 | — | — | — | — | — | — | 8 | 0.91 | 0.76 | 0.77 | 0.81 | |
| Chinchilla 70BNumber of shots=52022.04 | — | 67.5 | 63.6 | 54.9 | 79.3 | 73.9 | — | — | — | — | — | |
| HREestimator=HRE2026.06 | — | — | — | — | — | — | 11 | 0.87 | 0.72 | 0.71 | 0.77 | |
| NWKRestimator=NWKR2026.06 | — | — | — | — | — | — | 7 | 0.9 | 0.73 | 0.74 | 0.79 | |
| PaLM 540BNumber of shots=52022.04 | — | 69.3 | 77 | 55.6 | 81 | 69.6 | — | — | — | — | — | |
| PaLM 62BNumber of shots=52022.04 | — | 53.7 | 59.5 | 41.9 | 62.7 | 55.8 | — | — | — | — | — | |
| PaLM 8BNumber of shots=52022.04 | — | 25.3 | 25.6 | 23.8 | 24.1 | 27.8 | — | — | — | — | — | |
| Raw Confestimator=Raw Conf2026.06 | — | — | — | — | — | — | 17 | 0.9 | 0.71 | 0.54 | 0.72 |