Commonsense QA on HellaSwag
79.1AccuracyLlama 3-8B
Evaluation results
| Method | Links | |
|---|---|---|
| Llama 3-8BParadigm=AR, Training Tokens=15T, Training Data=Not Released, Evaluation protocol=Reported by prior work, Shots=02026.06 | 79.1 | |
| Falcon-7BParadigm=AR, Training Tokens=1.5T, Training Data=Partially Released, Evaluation protocol=Evaluated under our protocol, Shots=02026.06 | 76.3 | |
| Llama 2-7BParadigm=AR, Training Tokens=2T, Training Data=Not Released, Evaluation protocol=Evaluated under our protocol, Shots=02026.06 | 76.2 | |
| OLMo-7BParadigm=AR, Training Tokens=2.5T, Training Data=Fully Released, Evaluation protocol=Evaluated under our protocol, Shots=02026.06 | 75.6 | |
| LLaDA-8BParadigm=Masked Diffusion, Training Tokens=2.3T, Training Data=Not Released, Evaluation protocol=Reported by prior work, Shots=02026.06 | 70.5 | |
| YOCO (Dense)Model Scale=4B2026.06 | 67.6 | |
| YOCO (CLSA)Model Scale=4B, Attention=Cross-Layer Sparse2026.06 | 67.4 | |
| TransformerModel Scale=4B2026.06 | 66.7 | |
| Sumi-7BParadigm=Uniform Diffusion, Training Tokens=1.5T, Training Data=Fully Released, Evaluation protocol=Evaluated under our protocol, Shots=02026.06 | 60 | |
| FP16Backbone=LLaMA2-7B, Avg. Bit=16, Evaluation Protocol=zero-shot2024.10 | 57.1 | |
| CMPQBackbone=LLaMA2-7B, Avg. Bit=4, Evaluation Protocol=zero-shot2024.10 | 56.75 | |
| AWQBackbone=LLaMA2-7B, Avg. Bit=4, Evaluation Protocol=zero-shot2024.10 | 56.4 | |
| GPTQBackbone=LLaMA2-7B, Avg. Bit=4, Evaluation Protocol=zero-shot2024.10 | 55.99 | |
| QuIPBackbone=LLaMA2-7B, Avg. Bit=4, Evaluation Protocol=zero-shot2024.10 | 55.74 | |
| CMPQBackbone=LLaMA2-7B, Avg. Bit=3, Evaluation Protocol=zero-shot2024.10 | 53.85 | |
| RTNBackbone=LLaMA2-7B, Avg. Bit=4, Evaluation Protocol=zero-shot2024.10 | 53.7 | |
| QuIPBackbone=LLaMA2-7B, Avg. Bit=3, Evaluation Protocol=zero-shot2024.10 | 53.41 | |
| RTNBackbone=LLaMA2-7B, Avg. Bit=3, Evaluation Protocol=zero-shot2024.10 | 51.68 | |
| GPTQBackbone=LLaMA2-7B, Avg. Bit=3, Evaluation Protocol=zero-shot2024.10 | 49.05 |