Question Answering on Natural Questions (NQ) (5-shot)
38.8Accuracyfp16
Evaluation results
| Method | Links | |
|---|---|---|
| fp16Size=70B2026.02 | 38.8 | |
| OmniQuantSize=70B2026.02 | 36.1 | |
| AstroSize=70B2026.02 | 35.8 | |
| MagRSize=70B2026.02 | 35.7 | |
| fp16Size=13B2026.02 | 30.8 | |
| FP16Backbone=LLaMA-3.1-8B, Quantization=None, Protocol=5-shot2026.02 | 28.5 | |
| fp16Size=7B2026.02 | 26.7 | |
| MagRSize=13B2026.02 | 26.5 | |
| AstroSize=13B2026.02 | 26.5 | |
| OmniQuantSize=13B2026.02 | 25.5 | |
| AstroSize=7B2026.02 | 20.5 | |
| OmniQuantSize=7B2026.02 | 20.4 | |
| MagRSize=7B2026.02 | 20.4 | |
| AstroBackbone=LLaMA-3.1-8B, Quantization=W3A16g128, Protocol=5-shot2026.02 | 16.8 | |
| MagRBackbone=LLaMA-3.1-8B, Quantization=W3A16g128, Protocol=5-shot2026.02 | 15.8 | |
| OmniQuantBackbone=LLaMA-3.1-8B, Quantization=W3A16g128, Protocol=5-shot2026.02 | 14.8 |