# Reading Comprehension on BoolQ (Accuracy)

Source: [Reading Comprehension on BoolQ (Accuracy)](<https://www.sota2.com/research/sota/reading-comprehension-on-boolq-accuracy>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-24T07:05:05.289Z

Evaluation of Reading Comprehension performance on the BoolQ dataset, reporting Accuracy.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Reading Comprehension |
| Dataset (including any stated split) | BoolQ |
| Metric | Accuracy (BoolQ) |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 258 of 258 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Accuracy (BoolQ)](<https://www.sota2.com/research/sota/reading-comprehension-on-boolq-accuracy.md?metric=Accuracy%20(BoolQ)>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Accuracy (BoolQ) | Result date | Report paper |
| --- | --- | --- | --- |
| FP | 88.47 | Not recorded | 6a264c8290966ade919be92f |
| FP | 88.38 | Not recorded | 6a264c8290966ade919be92f |
| FAIR-Calib | 88.29 | Not recorded | 6a264c8290966ade919be92f |
| BEAM | 88.07 | Not recorded | 6a06a8acdbc836b87d82a1d5 |
| QuaRot | 87.98 | Not recorded | 6a264c8290966ade919be92f |
| FlatQuant | 87.98 | Not recorded | 6a264c8290966ade919be92f |
| FAIR-Calib | 87.89 | Not recorded | 6a264c8290966ade919be92f |
| Top-K Reduced | 87.68 | Not recorded | 6a06a8acdbc836b87d82a1d5 |
| MoE-Dynamic | 87.22 | Not recorded | 6a06a8acdbc836b87d82a1d5 |
| QuaRot | 87.07 | Not recorded | 6a264c8290966ade919be92f |
| FlatQuant | 86.92 | Not recorded | 6a264c8290966ade919be92f |
| Qwen3-30B-A3B | 86.76 | Not recorded | 6a06a8acdbc836b87d82a1d5 |
| Full-precision | 86.23 | Not recorded | 69d7327a7d0172140ec76a97 |
| HRM-Text 1B | 86.2 | Not recorded | 6a0e91a0f984739ed6738bdf |
| Uniform | 86.09 | Not recorded | 69d7327a7d0172140ec76a97 |
| RTN | 85.99 | Not recorded | 6a264c8290966ade919be92f |
| BF16 | 85.9 | Not recorded | 69fd6e395497b3b670c89547 |
| BEAM | 85.75 | Not recorded | 6a06a8acdbc836b87d82a1d5 |
| Router norm + Max var | 85.6 | Not recorded | 69d7327a7d0172140ec76a97 |
| Router norm + Max var | 85.57 | Not recorded | 69d7327a7d0172140ec76a97 |
| Olmo3 7B | 85.4 | Not recorded | 6a0e91a0f984739ed6738bdf |
| Router norm + Max var | 85.14 | Not recorded | 69d7327a7d0172140ec76a97 |
| Full-precision | 85.05 | Not recorded | 69d7327a7d0172140ec76a97 |
| Router norm + Max var | 84.92 | Not recorded | 69d7327a7d0172140ec76a97 |
| PMQ | 84.8 | Not recorded | 69d7327a7d0172140ec76a97 |
| AdaMoE | 84.71 | Not recorded | 6a06a8acdbc836b87d82a1d5 |
| RTN | 84.68 | Not recorded | 6a264c8290966ade919be92f |
| Router norm + Max var | 84.04 | Not recorded | 69d7327a7d0172140ec76a97 |
| Mistral-7B | 83.98 | Not recorded | 69d48f957d0172140ec74311 |
| Top-K Reduced | 83.85 | Not recorded | 6a06a8acdbc836b87d82a1d5 |
| PMQ | 83.67 | Not recorded | 69d7327a7d0172140ec76a97 |
| Ouro 1.4B | 83.6 | Not recorded | 6a0e91a0f984739ed6738bdf |
| PMQ | 83.15 | Not recorded | 69d7327a7d0172140ec76a97 |
| PMQ | 83.15 | Not recorded | 69d7327a7d0172140ec76a97 |
| SARQC-GBS(Saliency) | 82.6 | Not recorded | 69fd6e395497b3b670c89547 |
| SARQC-GBS(Identity) | 82.35 | Not recorded | 69fd6e395497b3b670c89547 |
| FP16 | 82.11 | Not recorded | 69fd6e395497b3b670c89547 |
| GPTQ | 82.11 | Not recorded | 69fd6e395497b3b670c89547 |
| Original | 82.02 | Not recorded | 6a06a89adbc836b87d82a06a |
| Original | 82.02 | Not recorded | 6a06a89adbc836b87d82a06a |
| PMQ | 81.93 | Not recorded | 69d7327a7d0172140ec76a97 |
| SARQC-GBS(Saliency) | 81.9 | Not recorded | 69fd6e395497b3b670c89547 |
| GPTQ | 81.25 | Not recorded | 69fd6e395497b3b670c89547 |
| Hessian | 81.11 | Not recorded | 69d7327a7d0172140ec76a97 |
| MoE-Dynamic | 81.07 | Not recorded | 6a06a8acdbc836b87d82a1d5 |
| Top-K Pruning | 81.04 | Not recorded | 6a06a8acdbc836b87d82a1d5 |
| Router norm + Max var | 80.61 | Not recorded | 69d7327a7d0172140ec76a97 |
| BF16 | 80.61 | Not recorded | 69fd6e395497b3b670c89547 |
| Qwen3.5 2B | 80.5 | Not recorded | 6a0e91a0f984739ed6738bdf |
| Router norm + Max var | 80.12 | Not recorded | 69d7327a7d0172140ec76a97 |
| SARQC-GBS(Identity) | 80.12 | Not recorded | 69fd6e395497b3b670c89547 |
| Unpruned | 80 | Not recorded | 69f03f4555a3c32af89fa24c |
| PMQ | 79.63 | Not recorded | 69d7327a7d0172140ec76a97 |
| FP16 | 79.27 | Not recorded | 69fd6e395497b3b670c89547 |
| Hessian | 79.15 | Not recorded | 69d7327a7d0172140ec76a97 |
| CoA-LoRA | 78.91 | Not recorded | 68fa7bb01e209131cea89e6e |
| PMQ | 78.65 | Not recorded | 69d7327a7d0172140ec76a97 |
| Uniform | 78.53 | Not recorded | 69d7327a7d0172140ec76a97 |
| BEAM | 78.47 | Not recorded | 6a06a8acdbc836b87d82a1d5 |
| Accuracy (Ours) | 78.38 | Not recorded | 6a06a89adbc836b87d82a06a |
| Shared-LoRA | 78.12 | Not recorded | 68fa7bb01e209131cea89e6e |
| Slice-GPT | 77.49 | Not recorded | 6a06a89adbc836b87d82a06a |
| Cosine Similarity | 76.7 | Not recorded | 6a06a89adbc836b87d82a06a |
| BEAM | 76.57 | Not recorded | 6a06a8acdbc836b87d82a1d5 |
| LQ-LoRA | 76.56 | Not recorded | 68fa7bb01e209131cea89e6e |
| Top-K Reduced | 76.39 | Not recorded | 6a06a8acdbc836b87d82a1d5 |
| Llama3.2 3B | 76.2 | Not recorded | 6a0e91a0f984739ed6738bdf |
| SARQC-GBS(Saliency) | 76.02 | Not recorded | 69fd6e395497b3b670c89547 |
| GPTQ | 75.84 | Not recorded | 69fd6e395497b3b670c89547 |
| SARQC-GBS(Identity) | 75.78 | Not recorded | 69fd6e395497b3b670c89547 |
| GPTAQ | 75.72 | Not recorded | 69fd6e395497b3b670c89547 |
| Cosine Similarity | 75.53 | Not recorded | 6a06a89adbc836b87d82a06a |
| DeepSeekV2-Lite | 75.2 | Not recorded | 6a06a8acdbc836b87d82a1d5 |
| QLoRA | 75 | Not recorded | 68fa7bb01e209131cea89e6e |
| OLMoE-1B-7B-0924 | 74.5 | Not recorded | 69cdf80e0636e5bf4afcca7a |
| PMQ | 74.28 | Not recorded | 69d7327a7d0172140ec76a97 |
| BF16 | 74.07 | Not recorded | 69fd6e395497b3b670c89547 |
| XPERT-OLMoE | 73.91 | Not recorded | 6a02b405dbc836b87d8245f3 |
| XPERT-OLMoE | 73.76 | Not recorded | 6a02b405dbc836b87d8245f3 |
| Router norm + Max var | 73.64 | Not recorded | 69d7327a7d0172140ec76a97 |

### FP

Accuracy (BoolQ): 88.47. Result ID: 6a265f94fbc373592c225e2f.

[Reporting paper](<https://www.sota2.com/research/paper/6a264c8290966ade919be92f.md>)

Recorded evaluation configuration: {"Model":"LLaDA-1.5","Quantization":"Full Precision"}. This record does not assert independent reproduction.

### FP

Accuracy (BoolQ): 88.38. Result ID: 6a265f94fbc373592c225e2a.

[Reporting paper](<https://www.sota2.com/research/paper/6a264c8290966ade919be92f.md>)

Recorded evaluation configuration: {"Model":"LLaDA-Instruct","Quantization":"Full Precision"}. This record does not assert independent reproduction.

### FAIR-Calib

Accuracy (BoolQ): 88.29. Result ID: 6a265f94fbc373592c225e2e.

[Reporting paper](<https://www.sota2.com/research/paper/6a264c8290966ade919be92f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a264c8290966ade919be92f.md>)

Recorded evaluation configuration: {"Model":"LLaDA-Instruct","Quantization":"W4A4"}. This record does not assert independent reproduction.

### BEAM

Accuracy (BoolQ): 88.07. Result ID: 6a06d7ba47f34ce924ee69f2.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

Recorded evaluation configuration: {"Sparsity Level":"Mid Sparsity","beta":"0.01"}. This record does not assert independent reproduction.

### QuaRot

Accuracy (BoolQ): 87.98. Result ID: 6a265f94fbc373592c225e2c.

[Reporting paper](<https://www.sota2.com/research/paper/6a264c8290966ade919be92f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa756a1e209131cea67b8c.md>)

[Implementation](<https://github.com/spcl/QuaRot>)

Recorded evaluation configuration: {"Model":"LLaDA-Instruct","Quantization":"W4A4"}. This record does not assert independent reproduction.

### FlatQuant

Accuracy (BoolQ): 87.98. Result ID: 6a265f94fbc373592c225e2d.

[Reporting paper](<https://www.sota2.com/research/paper/6a264c8290966ade919be92f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7a8b1e209131cea83cda.md>)

Recorded evaluation configuration: {"Model":"LLaDA-Instruct","Quantization":"W4A4"}. This record does not assert independent reproduction.

### FAIR-Calib

Accuracy (BoolQ): 87.89. Result ID: 6a265f94fbc373592c225e33.

[Reporting paper](<https://www.sota2.com/research/paper/6a264c8290966ade919be92f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a264c8290966ade919be92f.md>)

Recorded evaluation configuration: {"Model":"LLaDA-1.5","Quantization":"W4A4"}. This record does not assert independent reproduction.

### Top-K Reduced

Accuracy (BoolQ): 87.68. Result ID: 6a06d7ba47f34ce924ee69ef.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

Recorded evaluation configuration: {"Sparsity Level":"Mid Sparsity","K":"4"}. This record does not assert independent reproduction.

### MoE-Dynamic

Accuracy (BoolQ): 87.22. Result ID: 6a06d7ba47f34ce924ee69f0.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72811e209131cea56d72.md>)

Recorded evaluation configuration: {"Sparsity Level":"Mid Sparsity","phi":"0.3"}. This record does not assert independent reproduction.

### QuaRot

Accuracy (BoolQ): 87.07. Result ID: 6a265f94fbc373592c225e31.

[Reporting paper](<https://www.sota2.com/research/paper/6a264c8290966ade919be92f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa756a1e209131cea67b8c.md>)

[Implementation](<https://github.com/spcl/QuaRot>)

Recorded evaluation configuration: {"Model":"LLaDA-1.5","Quantization":"W4A4"}. This record does not assert independent reproduction.

### FlatQuant

Accuracy (BoolQ): 86.92. Result ID: 6a265f94fbc373592c225e32.

[Reporting paper](<https://www.sota2.com/research/paper/6a264c8290966ade919be92f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7a8b1e209131cea83cda.md>)

Recorded evaluation configuration: {"Model":"LLaDA-1.5","Quantization":"W4A4"}. This record does not assert independent reproduction.

### Qwen3-30B-A3B

Accuracy (BoolQ): 86.76. Result ID: 6a06d7ba47f34ce924ee69ed.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa788e1e209131cea79116.md>)

Recorded evaluation configuration: {"Sparsity Level":"Base","K":"8"}. This record does not assert independent reproduction.

### Full-precision

Accuracy (BoolQ): 86.23. Result ID: 69d8119747f34ce924bfa8f4.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x22B","Avg. bits/exp.":"16 (FP)","Memory (GB)":"281.2","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### HRM-Text 1B

Accuracy (BoolQ): 86.2. Result ID: 6a0ee7a5fbc373592c07097b.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e91a0f984739ed6738bdf.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a0e91a0f984739ed6738bdf.md>)

[Implementation](<https://github.com/sapientinc/HRM-Text>)

Recorded evaluation configuration: {"Architecture":"Recurrent","FLOPs ($10^{21}$)":"1","Tokens (T)":"0.06"}. This record does not assert independent reproduction.

### Uniform

Accuracy (BoolQ): 86.09. Result ID: 69d8119747f34ce924bfa8dd.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"3","Memory (GB)":"19.3","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### RTN

Accuracy (BoolQ): 85.99. Result ID: 6a265f94fbc373592c225e30.

[Reporting paper](<https://www.sota2.com/research/paper/6a264c8290966ade919be92f.md>)

Recorded evaluation configuration: {"Model":"LLaDA-1.5","Quantization":"W4A4"}. This record does not assert independent reproduction.

### BF16

Accuracy (BoolQ): 85.9. Result ID: 69fda1bd47f34ce924e0ae72.

[Reporting paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

Recorded evaluation configuration: {"Model":"Mixtral-8x7B","Precision":"BF16","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction.

### BEAM

Accuracy (BoolQ): 85.75. Result ID: 6a06d7ba47f34ce924ee69f7.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

Recorded evaluation configuration: {"Sparsity Level":"High Sparsity","beta":"0.1"}. This record does not assert independent reproduction.

### Router norm + Max var

Accuracy (BoolQ): 85.6. Result ID: 69d8119747f34ce924bfa8e0.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"2.625","Memory (GB)":"16.9","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### Router norm + Max var

Accuracy (BoolQ): 85.57. Result ID: 69d8119747f34ce924bfa8df.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"2.75","Memory (GB)":"17.7","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### Olmo3 7B

Accuracy (BoolQ): 85.4. Result ID: 6a0ee7a5fbc373592c07097d.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e91a0f984739ed6738bdf.md>)

[Method origin paper](<https://www.sota2.com/research/paper/694238e845614693628503e0.md>)

Recorded evaluation configuration: {"Architecture":"Dense","FLOPs ($10^{21}$)":"252","Tokens (T)":"6"}. This record does not assert independent reproduction.

### Router norm + Max var

Accuracy (BoolQ): 85.14. Result ID: 69d8119747f34ce924bfa8e1.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"2.5","Memory (GB)":"16.1","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### Full-precision

Accuracy (BoolQ): 85.05. Result ID: 69d8119747f34ce924bfa8dc.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"16 (FP)","Memory (GB)":"96.8","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### Router norm + Max var

Accuracy (BoolQ): 84.92. Result ID: 69d8119747f34ce924bfa8e2.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"2.375","Memory (GB)":"15.3","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### PMQ

Accuracy (BoolQ): 84.8. Result ID: 69d8119747f34ce924bfa8e7.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771f1e209131cea713b1.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"2.75","Memory (GB)":"17.7","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### AdaMoE

Accuracy (BoolQ): 84.71. Result ID: 6a06d7ba47f34ce924ee69f1.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75381e209131cea6687d.md>)

Recorded evaluation configuration: {"Sparsity Level":"Mid Sparsity","Null":"128"}. This record does not assert independent reproduction.

### RTN

Accuracy (BoolQ): 84.68. Result ID: 6a265f94fbc373592c225e2b.

[Reporting paper](<https://www.sota2.com/research/paper/6a264c8290966ade919be92f.md>)

Recorded evaluation configuration: {"Model":"LLaDA-Instruct","Quantization":"W4A4"}. This record does not assert independent reproduction.

### Router norm + Max var

Accuracy (BoolQ): 84.04. Result ID: 69d8119747f34ce924bfa8e3.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"2.25","Memory (GB)":"14.5","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### Mistral-7B

Accuracy (BoolQ): 83.98. Result ID: 69d4b3a447f34ce924bc2c18.

[Reporting paper](<https://www.sota2.com/research/paper/69d48f957d0172140ec74311.md>)

Recorded evaluation configuration: {"Ratio":"0%","Evaluation Protocol":"zero-shot"}. This record does not assert independent reproduction.

### Top-K Reduced

Accuracy (BoolQ): 83.85. Result ID: 6a06d7ba47f34ce924ee69f4.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

Recorded evaluation configuration: {"Sparsity Level":"High Sparsity","K":"2"}. This record does not assert independent reproduction.

### PMQ

Accuracy (BoolQ): 83.67. Result ID: 69d8119747f34ce924bfa8e8.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771f1e209131cea713b1.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"2.625","Memory (GB)":"16.9","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### Ouro 1.4B

Accuracy (BoolQ): 83.6. Result ID: 6a0ee7a5fbc373592c070981.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e91a0f984739ed6738bdf.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6902f1003ef7ccd74e5e4419.md>)

Recorded evaluation configuration: {"Architecture":"Recurrent","FLOPs ($10^{21}$)":"259","Tokens (T)":"7"}. This record does not assert independent reproduction.

### PMQ

Accuracy (BoolQ): 83.15. Result ID: 69d8119747f34ce924bfa8e9.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771f1e209131cea713b1.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"2.5","Memory (GB)":"16.1","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### PMQ

Accuracy (BoolQ): 83.15. Result ID: 69d8119747f34ce924bfa8eb.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771f1e209131cea713b1.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"2.25","Memory (GB)":"14.5","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### SARQC-GBS(Saliency)

Accuracy (BoolQ): 82.6. Result ID: 69fda1bd47f34ce924e0ae75.

[Reporting paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

Recorded evaluation configuration: {"Model":"Mixtral-8x7B","Precision":"W3A16","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction.

### SARQC-GBS(Identity)

Accuracy (BoolQ): 82.35. Result ID: 69fda1bd47f34ce924e0ae74.

[Reporting paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

Recorded evaluation configuration: {"Model":"Mixtral-8x7B","Precision":"W3A16","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction.

### FP16

Accuracy (BoolQ): 82.11. Result ID: 69fda1bd47f34ce924e0ae65.

[Reporting paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

Recorded evaluation configuration: {"Model":"LLaMA2-13B","Precision":"FP16","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction.

### GPTQ

Accuracy (BoolQ): 82.11. Result ID: 69fda1bd47f34ce924e0ae73.

[Reporting paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6ecc1e209131cea4235d.md>)

Recorded evaluation configuration: {"Model":"Mixtral-8x7B","Precision":"W3A16","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction.

### Original

Accuracy (BoolQ): 82.02. Result ID: 6a06cd3247f34ce924ee1598.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

Recorded evaluation configuration: {"Base Model":"LLaMA3-8B"}. This record does not assert independent reproduction.

### Original

Accuracy (BoolQ): 82.02. Result ID: 6a06cd3247f34ce924ee15de.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA3-8B","Pruning Ratio":"0%"}. This record does not assert independent reproduction.

### PMQ

Accuracy (BoolQ): 81.93. Result ID: 69d8119747f34ce924bfa8ea.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771f1e209131cea713b1.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"2.375","Memory (GB)":"15.3","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### SARQC-GBS(Saliency)

Accuracy (BoolQ): 81.9. Result ID: 69fda1bd47f34ce924e0ae71.

[Reporting paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

Recorded evaluation configuration: {"Model":"Qwen3-MoE-30B","Precision":"W3A16","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction.

### GPTQ

Accuracy (BoolQ): 81.25. Result ID: 69fda1bd47f34ce924e0ae6f.

[Reporting paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6ecc1e209131cea4235d.md>)

Recorded evaluation configuration: {"Model":"Qwen3-MoE-30B","Precision":"W3A16","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction.

### Hessian

Accuracy (BoolQ): 81.11. Result ID: 69d8119747f34ce924bfa8ef.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa69e51e209131cea24aa3.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"2.5","Memory (GB)":"17.0","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### MoE-Dynamic

Accuracy (BoolQ): 81.07. Result ID: 6a06d7ba47f34ce924ee69f5.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72811e209131cea56d72.md>)

Recorded evaluation configuration: {"Sparsity Level":"High Sparsity","phi":"0.1"}. This record does not assert independent reproduction.

### Top-K Pruning

Accuracy (BoolQ): 81.04. Result ID: 6a06d7ba47f34ce924ee69ee.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

Recorded evaluation configuration: {"Sparsity Level":"Mid Sparsity","K":"4"}. This record does not assert independent reproduction.

### Router norm + Max var

Accuracy (BoolQ): 80.61. Result ID: 69d8119747f34ce924bfa8e5.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"2.0","Memory (GB)":"13.1","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### BF16

Accuracy (BoolQ): 80.61. Result ID: 69fda1bd47f34ce924e0ae6e.

[Reporting paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

Recorded evaluation configuration: {"Model":"Qwen3-MoE-30B","Precision":"BF16","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction.

### Qwen3.5 2B

Accuracy (BoolQ): 80.5. Result ID: 6a0ee7a5fbc373592c070980.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e91a0f984739ed6738bdf.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa788e1e209131cea79116.md>)

Recorded evaluation configuration: {"Architecture":"Dense","FLOPs ($10^{21}$)":"432","Tokens (T)":"36"}. This record does not assert independent reproduction.

### Router norm + Max var

Accuracy (BoolQ): 80.12. Result ID: 69d8119747f34ce924bfa8e4.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"2.125","Memory (GB)":"13.8","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### SARQC-GBS(Identity)

Accuracy (BoolQ): 80.12. Result ID: 69fda1bd47f34ce924e0ae70.

[Reporting paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

Recorded evaluation configuration: {"Model":"Qwen3-MoE-30B","Precision":"W3A16","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction.

### Unpruned

Accuracy (BoolQ): 80. Result ID: 69f0707b47f34ce924d659cf.

[Reporting paper](<https://www.sota2.com/research/paper/69f03f4555a3c32af89fa24c.md>)

Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B"}. This record does not assert independent reproduction.

### PMQ

Accuracy (BoolQ): 79.63. Result ID: 69d8119747f34ce924bfa8ee.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771f1e209131cea713b1.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"1.75","Memory (GB)":"11.7","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### FP16

Accuracy (BoolQ): 79.27. Result ID: 69fda1bd47f34ce924e0ae60.

[Reporting paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

Recorded evaluation configuration: {"Model":"LLaMA2-7B","Precision":"FP16","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction.

### Hessian

Accuracy (BoolQ): 79.15. Result ID: 69d8119747f34ce924bfa8f0.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa69e51e209131cea24aa3.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"2.25","Memory (GB)":"15.3","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### CoA-LoRA

Accuracy (BoolQ): 78.91. Result ID: 69dc988d47f34ce924c32bfa.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bb01e209131cea89e6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bb01e209131cea89e6e.md>)

Recorded evaluation configuration: {"Model":"Llama-2-7B","Average Bit Budget":"3-bit","Zero-shot":"true"}. This record does not assert independent reproduction.

### PMQ

Accuracy (BoolQ): 78.65. Result ID: 69d8119747f34ce924bfa8ed.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771f1e209131cea713b1.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"2.0","Memory (GB)":"13.1","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### Uniform

Accuracy (BoolQ): 78.53. Result ID: 69d8119747f34ce924bfa8f5.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x22B","Avg. bits/exp.":"3","Memory (GB)":"57.5","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### BEAM

Accuracy (BoolQ): 78.47. Result ID: 6a06d7ba47f34ce924ee69f9.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

Recorded evaluation configuration: {"Sparsity Level":"Extreme Sparsity","beta":"1.0"}. This record does not assert independent reproduction.

### Accuracy (Ours)

Accuracy (BoolQ): 78.38. Result ID: 6a06cd3247f34ce924ee15e6.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA3-8B","Pruning Ratio":"25%"}. This record does not assert independent reproduction.

### Shared-LoRA

Accuracy (BoolQ): 78.12. Result ID: 69dc988d47f34ce924c32bf9.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bb01e209131cea89e6e.md>)

Recorded evaluation configuration: {"Model":"Llama-2-7B","Average Bit Budget":"3-bit","Zero-shot":"true"}. This record does not assert independent reproduction.

### Slice-GPT

Accuracy (BoolQ): 77.49. Result ID: 6a06cd3247f34ce924ee15e5.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72271e209131cea54d13.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA3-8B","Pruning Ratio":"25%"}. This record does not assert independent reproduction.

### Cosine Similarity

Accuracy (BoolQ): 76.7. Result ID: 6a06cd3247f34ce924ee159a.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75401e209131cea66d5a.md>)

Recorded evaluation configuration: {"Pruning Method":"Cosine Similarity","Base Model":"LLaMA3-8B","Calibration Dataset Size":"1,500-instance"}. This record does not assert independent reproduction.

### BEAM

Accuracy (BoolQ): 76.57. Result ID: 6a06d7ba47f34ce924ee697e.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

Recorded evaluation configuration: {"Sparsity Level":"Mid Sparsity","beta":"0.01","Avg. K":"2.61"}. This record does not assert independent reproduction.

### LQ-LoRA

Accuracy (BoolQ): 76.56. Result ID: 69dc988d47f34ce924c32bf8.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bb01e209131cea89e6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa74a51e209131cea62e31.md>)

Recorded evaluation configuration: {"Model":"Llama-2-7B","Average Bit Budget":"3-bit","Zero-shot":"true"}. This record does not assert independent reproduction.

### Top-K Reduced

Accuracy (BoolQ): 76.39. Result ID: 6a06d7ba47f34ce924ee697b.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

Recorded evaluation configuration: {"Sparsity Level":"Mid Sparsity","K":"4","Avg. K":"4.00"}. This record does not assert independent reproduction.

### Llama3.2 3B

Accuracy (BoolQ): 76.2. Result ID: 6a0ee7a5fbc373592c07097e.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e91a0f984739ed6738bdf.md>)

Recorded evaluation configuration: {"Architecture":"Dense","FLOPs ($10^{21}$)":"162","Tokens (T)":"9"}. This record does not assert independent reproduction.

### SARQC-GBS(Saliency)

Accuracy (BoolQ): 76.02. Result ID: 69fda1bd47f34ce924e0ae69.

[Reporting paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

Recorded evaluation configuration: {"Model":"LLaMA2-13B","Precision":"W3A16","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction.

### GPTQ

Accuracy (BoolQ): 75.84. Result ID: 69fda1bd47f34ce924e0ae66.

[Reporting paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6ecc1e209131cea4235d.md>)

Recorded evaluation configuration: {"Model":"LLaMA2-13B","Precision":"W3A16","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction.

### SARQC-GBS(Identity)

Accuracy (BoolQ): 75.78. Result ID: 69fda1bd47f34ce924e0ae68.

[Reporting paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

Recorded evaluation configuration: {"Model":"LLaMA2-13B","Precision":"W3A16","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction.

### GPTAQ

Accuracy (BoolQ): 75.72. Result ID: 69fda1bd47f34ce924e0ae67.

[Reporting paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa788d1e209131cea79026.md>)

Recorded evaluation configuration: {"Model":"LLaMA2-13B","Precision":"W3A16","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction.

### Cosine Similarity

Accuracy (BoolQ): 75.53. Result ID: 6a06cd3247f34ce924ee15e0.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75401e209131cea66d5a.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA3-8B","Pruning Ratio":"25%"}. This record does not assert independent reproduction.

### DeepSeekV2-Lite

Accuracy (BoolQ): 75.2. Result ID: 6a06d7ba47f34ce924ee6979.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a8acdbc836b87d82a1d5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73b81e209131cea5e189.md>)

Recorded evaluation configuration: {"K":"6","Avg. K":"6.00"}. This record does not assert independent reproduction.

### QLoRA

Accuracy (BoolQ): 75. Result ID: 69dc988d47f34ce924c32bf7.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bb01e209131cea89e6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f6a1e209131cea456e1.md>)

[Implementation](<https://github.com/artidoro/qlora>)

Recorded evaluation configuration: {"Model":"Llama-2-7B","Average Bit Budget":"3-bit","Zero-shot":"true"}. This record does not assert independent reproduction.

### OLMoE-1B-7B-0924

Accuracy (BoolQ): 74.5. Result ID: 69ce3dd447f34ce924b7e838.

[Reporting paper](<https://www.sota2.com/research/paper/69cdf80e0636e5bf4afcca7a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa773e1e209131cea71f14.md>)

Recorded evaluation configuration: {"Model Type":"MoE"}. This record does not assert independent reproduction.

### PMQ

Accuracy (BoolQ): 74.28. Result ID: 69d8119747f34ce924bfa8fb.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771f1e209131cea713b1.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x22B","Avg. bits/exp.":"2.5","Memory (GB)":"46.7","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

### BF16

Accuracy (BoolQ): 74.07. Result ID: 69fda1bd47f34ce924e0ae6a.

[Reporting paper](<https://www.sota2.com/research/paper/69fd6e395497b3b670c89547.md>)

Recorded evaluation configuration: {"Model":"DeepSeek-MoE-16B","Precision":"BF16","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction.

### XPERT-OLMoE

Accuracy (BoolQ): 73.91. Result ID: 6a02f90147f34ce924e637f8.

[Reporting paper](<https://www.sota2.com/research/paper/6a02b405dbc836b87d8245f3.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a02b405dbc836b87d8245f3.md>)

Recorded evaluation configuration: {"#Params":"570M","#Size":"dim=2048 16 Layers","Pre-training Tokens":"5B","Fine-tuning Epochs":"3"}. This record does not assert independent reproduction.

### XPERT-OLMoE

Accuracy (BoolQ): 73.76. Result ID: 6a02f90147f34ce924e637f4.

[Reporting paper](<https://www.sota2.com/research/paper/6a02b405dbc836b87d8245f3.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a02b405dbc836b87d8245f3.md>)

Recorded evaluation configuration: {"#Params":"480M","#Size":"dim=2048 12 Layers","Pre-training Tokens":"5B","Fine-tuning Epochs":"3"}. This record does not assert independent reproduction.

### Router norm + Max var

Accuracy (BoolQ): 73.64. Result ID: 69d8119747f34ce924bfa8e6.

[Reporting paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69d7327a7d0172140ec76a97.md>)

Recorded evaluation configuration: {"Model":"Mixtral 8x7B","Avg. bits/exp.":"1.75","Memory (GB)":"11.7","Evaluation protocol":"zero-shot"}. This record does not assert independent reproduction.

Page 1 of 4. This is an index page, not the full collection.

- [Next index page](<https://www.sota2.com/research/sota/reading-comprehension-on-boolq-accuracy.md?metric=Accuracy%20(BoolQ)&page=2>): Follow this link to discover more pages.
