# Question Answering on OpenBookQA (Accuracy and Performance Gain)

Source: [Question Answering on OpenBookQA (Accuracy and Performance Gain)](<https://www.sota2.com/research/sota/question-answering-on-openbookqa-accuracy-and-performance-gain>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-09T07:17:36.605Z

Evaluation of model performance on the OpenBookQA dataset for Question Answering, measured using Accuracy and Performance Gain.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Question Answering |
| Dataset (including any stated split) | OpenBookQA |
| Metric | Accuracy |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 157 of 157 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Accuracy](<https://www.sota2.com/research/sota/question-answering-on-openbookqa-accuracy-and-performance-gain.md?metric=Accuracy>): Higher is better; unit: PERCENT.

- [Performance Gain](<https://www.sota2.com/research/sota/question-answering-on-openbookqa-accuracy-and-performance-gain.md?metric=Performance%20Gain>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Accuracy | Result date | Report paper |
| --- | --- | --- | --- |
| Qwen3 8B Base | 84.4 | Not recorded | 69802f6da8b01e76cc8e150e |
| Qwen3 8B Base | 84.2 | Not recorded | 69802f6da8b01e76cc8e150e |
| Kimi-Audio | 83.5 | Not recorded | 6a4f2b2b32fb937314875d39 |
| TS (C) | 79.3 | Not recorded | 699e820b6017c4460ffacf00 |
| Llama 3.1 8B | 76.6 | Not recorded | 69802f6da8b01e76cc8e150e |
| Llama 3.1 8B | 76.6 | Not recorded | 69802f6da8b01e76cc8e150e |
| DuplexPO | 73.7 | Not recorded | 6a4f2b2b32fb937314875d39 |
| Mistral Nemo Base 2407 | 73.4 | Not recorded | 69802f6da8b01e76cc8e150e |
| Mistral Nemo Base 2407 | 73 | Not recorded | 69802f6da8b01e76cc8e150e |
| SFT Baseline | 72.2 | Not recorded | 6a4f2b2b32fb937314875d39 |
| Baichuan-Audio | 71.7 | Not recorded | 6a4f2b2b32fb937314875d39 |
| Mistral 7B v0.3 | 68.8 | Not recorded | 69802f6da8b01e76cc8e150e |
| Mistral 7B v0.3 | 68.4 | Not recorded | 69802f6da8b01e76cc8e150e |
| GLM-4-Voice | 53.4 | Not recorded | 6a4f2b2b32fb937314875d39 |
| Qwen3 0.6B | 53 | Not recorded | 69802f6da8b01e76cc8e150e |
| Qwen3 0.6B | 52.2 | Not recorded | 69802f6da8b01e76cc8e150e |
| Qwen2-Audio | 49.5 | Not recorded | 6a4f2b2b32fb937314875d39 |
| Llama 3.2 1B | 47.4 | Not recorded | 69802f6da8b01e76cc8e150e |
| Moonlight | 45.6 | Not recorded | 68fa79ce1e209131cea7f78a |
| Qwen3-30B-A3B | 44.8 | Not recorded | 68fa79ce1e209131cea7f78a |
| Original | 44.8 | Not recorded | 6a06a89adbc836b87d82a06a |
| Original | 44.8 | Not recorded | 6a06a89adbc836b87d82a06a |
| OLMoE | 44.4 | Not recorded | 68fa79ce1e209131cea7f78a |
| Qwen2-57B-A14B | 44.2 | Not recorded | 68fa79ce1e209131cea7f78a |
| Llama 3.2 1B | 44 | Not recorded | 69802f6da8b01e76cc8e150e |
| Clean | 44 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| Fine-Pruning | 43.8 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| BD-VAX | 43.8 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| Accuracy (Ours) | 43.8 | Not recorded | 6a06a89adbc836b87d82a06a |
| Deepseek-V2-Lite | 43.6 | Not recorded | 68fa79ce1e209131cea7f78a |
| Clean | 43.6 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| FT | 43.6 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| Quantization | 43.6 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| CROW | 43.6 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| Accuracy (Arc-E) | 43.6 | Not recorded | 6a06a89adbc836b87d82a06a |
| Fine-Pruning | 43.4 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| Fine-Pruning | 43 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| BD-VAX | 43 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| Baichuan-7b | 42.8 | Not recorded | 68fa76891e209131cea6d801 |
| RS | 42.4 | Not recorded | 68fa79ce1e209131cea7f78a |
| Fine-Pruning | 42.4 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| BD-VAX | 42.4 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| FT | 42.2 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| Pruning | 42.2 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| Pruning | 42.2 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| FT | 42.16 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| MoNE | 42 | Not recorded | 68fa79ce1e209131cea7f78a |
| Attacked | 42 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| MoNE | 41.6 | Not recorded | 68fa79ce1e209131cea7f78a |
| Ettin-Dec-1B | 41.6 | Not recorded | 68fa7a111e209131cea80ee7 |
| Attacked | 41.4 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| Taylor | 41.4 | Not recorded | 6a06a89adbc836b87d82a06a |
| Out. Norm-Sim | 41.4 | Not recorded | 6a06a89adbc836b87d82a06a |
| Taylor | 41.4 | Not recorded | 6a06a89adbc836b87d82a06a |
| FT | 41.2 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| MoNE | 40.8 | Not recorded | 68fa79ce1e209131cea7f78a |
| RS | 40.8 | Not recorded | 68fa79ce1e209131cea7f78a |
| BD-VAX | 40.6 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| CROW | 40.2 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| Out. Cosine-Sim | 40.2 | Not recorded | 6a06a89adbc836b87d82a06a |
| Pruning | 40 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| HyWIA | 39.6 | Not recorded | 68fa76891e209131cea6d801 |
| Out. Divergence-Sim | 39.6 | Not recorded | 6a06a89adbc836b87d82a06a |
| Out. Norm-Sim | 39.6 | Not recorded | 6a06a89adbc836b87d82a06a |
| SALM-Duplex | 39.6 | Not recorded | 6a4f2b2b32fb937314875d39 |
| MC-SMoE | 39.4 | Not recorded | 68fa79ce1e209131cea7f78a |
| Quantization | 39.4 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| Perplexity | 39.2 | Not recorded | 6a06a89adbc836b87d82a06a |
| MC-SMoE | 38.8 | Not recorded | 68fa79ce1e209131cea7f78a |
| LLM-Pruner Vector⋆ | 38.4 | Not recorded | 68fa76891e209131cea6d801 |
| FLAP | 38.2 | Not recorded | 68fa79ce1e209131cea7f78a |
| RS | 38 | Not recorded | 68fa79ce1e209131cea7f78a |
| LLM-Pruner Element2⋆ | 37.8 | Not recorded | 68fa76891e209131cea6d801 |
| SmolLM2-360m | 37.6 | Not recorded | 68fa7a111e209131cea80ee7 |
| Pruning | 37.4 | Not recorded | 69b8e00c0636e5bf4afbea41 |
| Llama-3.2-1B | 37.2 | Not recorded | 68fa7a111e209131cea80ee7 |
| MoNE | 36.8 | Not recorded | 68fa79ce1e209131cea7f78a |
| Cosine Similarity | 36.8 | Not recorded | 6a06a89adbc836b87d82a06a |
| Slice-GPT | 36.8 | Not recorded | 6a06a89adbc836b87d82a06a |
| Cosine Similarity | 36.2 | Not recorded | 6a06a89adbc836b87d82a06a |

### Qwen3 8B Base

Accuracy: 84.4. Result ID: 699ab11f5698665ec43e969d.

[Reporting paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

Recorded evaluation configuration: {"Classifier":"Majority-labeled"}. This record does not assert independent reproduction.

### Qwen3 8B Base

Accuracy: 84.2. Result ID: 699ab11f5698665ec43e969c.

[Reporting paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

Recorded evaluation configuration: {"Classifier":"Self-labeled"}. This record does not assert independent reproduction.

### Kimi-Audio

Accuracy: 83.5. Result ID: 6a4f4b0360b88bb339e4bae9.

[Reporting paper](<https://www.sota2.com/research/paper/6a4f2b2b32fb937314875d39.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78431e209131cea77803.md>)

Recorded evaluation configuration: {"Full Duplex":"false"}. This record does not assert independent reproduction.

### TS (C)

Accuracy: 79.3. Result ID: 699eaa375698665ec4766f48.

[Reporting paper](<https://www.sota2.com/research/paper/699e820b6017c4460ffacf00.md>)

[Method origin paper](<https://www.sota2.com/research/paper/699e820b6017c4460ffacf00.md>)

Recorded evaluation configuration: {"Contextual":true}. This record does not assert independent reproduction.

### Llama 3.1 8B

Accuracy: 76.6. Result ID: 699ab11f5698665ec43e9694.

[Reporting paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

Recorded evaluation configuration: {"Classifier":"Self-labeled"}. This record does not assert independent reproduction.

### Llama 3.1 8B

Accuracy: 76.6. Result ID: 699ab11f5698665ec43e9695.

[Reporting paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

Recorded evaluation configuration: {"Classifier":"Majority-labeled"}. This record does not assert independent reproduction.

### DuplexPO

Accuracy: 73.7. Result ID: 6a4f4b0360b88bb339e4baec.

[Reporting paper](<https://www.sota2.com/research/paper/6a4f2b2b32fb937314875d39.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a4f2b2b32fb937314875d39.md>)

Recorded evaluation configuration: {"Full Duplex":"true"}. This record does not assert independent reproduction.

### Mistral Nemo Base 2407

Accuracy: 73.4. Result ID: 699ab11f5698665ec43e9699.

[Reporting paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

Recorded evaluation configuration: {"Classifier":"Majority-labeled"}. This record does not assert independent reproduction.

### Mistral Nemo Base 2407

Accuracy: 73. Result ID: 699ab11f5698665ec43e9698.

[Reporting paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

Recorded evaluation configuration: {"Classifier":"Self-labeled"}. This record does not assert independent reproduction.

### SFT Baseline

Accuracy: 72.2. Result ID: 6a4f4b0360b88bb339e4baeb.

[Reporting paper](<https://www.sota2.com/research/paper/6a4f2b2b32fb937314875d39.md>)

Recorded evaluation configuration: {"Full Duplex":"true"}. This record does not assert independent reproduction.

### Baichuan-Audio

Accuracy: 71.7. Result ID: 6a4f4b0360b88bb339e4baea.

[Reporting paper](<https://www.sota2.com/research/paper/6a4f2b2b32fb937314875d39.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771a1e209131cea71007.md>)

Recorded evaluation configuration: {"Full Duplex":"false"}. This record does not assert independent reproduction.

### Mistral 7B v0.3

Accuracy: 68.8. Result ID: 699ab11f5698665ec43e9696.

[Reporting paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

Recorded evaluation configuration: {"Classifier":"Self-labeled"}. This record does not assert independent reproduction.

### Mistral 7B v0.3

Accuracy: 68.4. Result ID: 699ab11f5698665ec43e9697.

[Reporting paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

Recorded evaluation configuration: {"Classifier":"Majority-labeled"}. This record does not assert independent reproduction.

### GLM-4-Voice

Accuracy: 53.4. Result ID: 6a4f4b0360b88bb339e4bae7.

[Reporting paper](<https://www.sota2.com/research/paper/6a4f2b2b32fb937314875d39.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75f91e209131cea6abba.md>)

Recorded evaluation configuration: {"Full Duplex":"false"}. This record does not assert independent reproduction.

### Qwen3 0.6B

Accuracy: 53. Result ID: 699ab11f5698665ec43e969a.

[Reporting paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

Recorded evaluation configuration: {"Classifier":"Self-labeled"}. This record does not assert independent reproduction.

### Qwen3 0.6B

Accuracy: 52.2. Result ID: 699ab11f5698665ec43e969b.

[Reporting paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

Recorded evaluation configuration: {"Classifier":"Majority-labeled"}. This record does not assert independent reproduction.

### Qwen2-Audio

Accuracy: 49.5. Result ID: 6a4f4b0360b88bb339e4bae8.

[Reporting paper](<https://www.sota2.com/research/paper/6a4f2b2b32fb937314875d39.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa74281e209131cea6017e.md>)

Recorded evaluation configuration: {"Full Duplex":"false"}. This record does not assert independent reproduction.

### Llama 3.2 1B

Accuracy: 47.4. Result ID: 699ab11f5698665ec43e9692.

[Reporting paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

Recorded evaluation configuration: {"Classifier":"Self-labeled"}. This record does not assert independent reproduction.

### Moonlight

Accuracy: 45.6. Result ID: 699e19595698665ec4d23e3e.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771a1e209131cea70fcd.md>)

Recorded evaluation configuration: {"Pruning ratio":"0%","Base Model":"Moonlight","Evaluation protocol":"Zero-shot","Calibration samples":100,"Calibration dataset":"Zyda2"}. This record does not assert independent reproduction.

### Qwen3-30B-A3B

Accuracy: 44.8. Result ID: 699e19595698665ec4d23e50.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa788e1e209131cea79116.md>)

Recorded evaluation configuration: {"Pruning ratio":"0%","Base Model":"Qwen3-30B-A3B","Evaluation protocol":"Zero-shot","Calibration samples":100,"Calibration dataset":"Zyda2"}. This record does not assert independent reproduction.

### Original

Accuracy: 44.8. Result ID: 6a06cd3247f34ce924ee15a8.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

Recorded evaluation configuration: {"Base Model":"LLaMA3-8B"}. This record does not assert independent reproduction.

### Original

Accuracy: 44.8. Result ID: 6a06cd3247f34ce924ee15f4.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA3-8B"}. This record does not assert independent reproduction.

### OLMoE

Accuracy: 44.4. Result ID: 699e19595698665ec4d23e38.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa773e1e209131cea71f14.md>)

Recorded evaluation configuration: {"Pruning ratio":"0%","Base Model":"OLMoE","Evaluation protocol":"Zero-shot","Calibration samples":100,"Calibration dataset":"Zyda2"}. This record does not assert independent reproduction.

### Qwen2-57B-A14B

Accuracy: 44.2. Result ID: 699e19595698665ec4d23e4a.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa74bf1e209131cea6378d.md>)

[Implementation](<https://github.com/QwenLM/Qwen2>)

Recorded evaluation configuration: {"Pruning ratio":"0%","Base Model":"Qwen2-57B-A14B","Evaluation protocol":"Zero-shot","Calibration samples":100,"Calibration dataset":"Zyda2"}. This record does not assert independent reproduction.

### Llama 3.2 1B

Accuracy: 44. Result ID: 699ab11f5698665ec43e9693.

[Reporting paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69802f6da8b01e76cc8e150e.md>)

Recorded evaluation configuration: {"Classifier":"Majority-labeled"}. This record does not assert independent reproduction.

### Clean

Accuracy: 44. Result ID: 69b90bf447f34ce924a16ba4.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-13B-Chat"}. This record does not assert independent reproduction.

### Fine-Pruning

Accuracy: 43.8. Result ID: 69b90bf447f34ce924a16bae.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa646b1e209131cea04983.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-13B-Chat","Parameter Setting":"LoRA Adapter"}. This record does not assert independent reproduction.

### BD-VAX

Accuracy: 43.8. Result ID: 69b90bf447f34ce924a16baf.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-13B-Chat","Parameter Setting":"LoRA Adapter"}. This record does not assert independent reproduction.

### Accuracy (Ours)

Accuracy: 43.8. Result ID: 6a06cd3247f34ce924ee15fc.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA3-8B","Pruning Ratio":"25%"}. This record does not assert independent reproduction.

### Deepseek-V2-Lite

Accuracy: 43.6. Result ID: 699e19595698665ec4d23e44.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73b81e209131cea5e189.md>)

Recorded evaluation configuration: {"Pruning ratio":"0%","Base Model":"Deepseek-V2-Lite","Evaluation protocol":"Zero-shot","Calibration samples":100,"Calibration dataset":"Zyda2"}. This record does not assert independent reproduction.

### Clean

Accuracy: 43.6. Result ID: 69b90bf447f34ce924a16b98.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-7B-Chat"}. This record does not assert independent reproduction.

### FT

Accuracy: 43.6. Result ID: 69b90bf447f34ce924a16ba6.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa70d01e209131cea4d60a.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-13B-Chat","Parameter Setting":"Full Params"}. This record does not assert independent reproduction.

### Quantization

Accuracy: 43.6. Result ID: 69b90bf447f34ce924a16ba8.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa67901e209131cea170e6.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-13B-Chat","Parameter Setting":"Full Params"}. This record does not assert independent reproduction.

### CROW

Accuracy: 43.6. Result ID: 69b90bf447f34ce924a16ba9.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa795d1e209131cea7d47c.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-13B-Chat","Parameter Setting":"Full Params"}. This record does not assert independent reproduction.

### Accuracy (Arc-E)

Accuracy: 43.6. Result ID: 6a06cd3247f34ce924ee15fd.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA3-8B","Pruning Ratio":"25%"}. This record does not assert independent reproduction.

### Fine-Pruning

Accuracy: 43.4. Result ID: 69b90bf447f34ce924a16baa.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa646b1e209131cea04983.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-13B-Chat","Parameter Setting":"Full Params"}. This record does not assert independent reproduction.

### Fine-Pruning

Accuracy: 43. Result ID: 69b90bf447f34ce924a16b9e.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa646b1e209131cea04983.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-7B-Chat","Parameter Setting":"Full Params"}. This record does not assert independent reproduction.

### BD-VAX

Accuracy: 43. Result ID: 69b90bf447f34ce924a16bab.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-13B-Chat","Parameter Setting":"Full Params"}. This record does not assert independent reproduction.

### Baichuan-7b

Accuracy: 42.8. Result ID: 69b272446dbb7075b95cd922.

[Reporting paper](<https://www.sota2.com/research/paper/68fa76891e209131cea6d801.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78081e209131cea766ec.md>)

Recorded evaluation configuration: {"Ratio":"0%","Zero-shot":"true"}. This record does not assert independent reproduction.

### RS

Accuracy: 42.4. Result ID: 699e19595698665ec4d23e54.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa777b1e209131cea7376a.md>)

Recorded evaluation configuration: {"Pruning ratio":"50%","Base Model":"Qwen3-30B-A3B","Evaluation protocol":"Zero-shot","Calibration samples":100,"Calibration dataset":"Zyda2"}. This record does not assert independent reproduction.

### Fine-Pruning

Accuracy: 42.4. Result ID: 69b90bf447f34ce924a16ba2.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa646b1e209131cea04983.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-7B-Chat","Parameter Setting":"LoRA Adapter"}. This record does not assert independent reproduction.

### BD-VAX

Accuracy: 42.4. Result ID: 69b90bf447f34ce924a16ba3.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-7B-Chat","Parameter Setting":"LoRA Adapter"}. This record does not assert independent reproduction.

### FT

Accuracy: 42.2. Result ID: 69b90bf447f34ce924a16b9a.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa70d01e209131cea4d60a.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-7B-Chat","Parameter Setting":"Full Params"}. This record does not assert independent reproduction.

### Pruning

Accuracy: 42.2. Result ID: 69b90bf447f34ce924a16ba1.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6b181e209131cea2c1a4.md>)

[Implementation](<https://github.com/csdongxian/ANP_backdoor>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-7B-Chat","Parameter Setting":"LoRA Adapter"}. This record does not assert independent reproduction.

### Pruning

Accuracy: 42.2. Result ID: 69b90bf447f34ce924a16bad.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6b181e209131cea2c1a4.md>)

[Implementation](<https://github.com/csdongxian/ANP_backdoor>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-13B-Chat","Parameter Setting":"LoRA Adapter"}. This record does not assert independent reproduction.

### FT

Accuracy: 42.16. Result ID: 69b90bf447f34ce924a16bac.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa70d01e209131cea4d60a.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-13B-Chat","Parameter Setting":"LoRA Adapter"}. This record does not assert independent reproduction.

### MoNE

Accuracy: 42. Result ID: 699e19595698665ec4d23e4f.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

Recorded evaluation configuration: {"Pruning ratio":"50%","Base Model":"Qwen2-57B-A14B","Evaluation protocol":"Zero-shot","Calibration samples":100,"Calibration dataset":"Zyda2"}. This record does not assert independent reproduction.

### Attacked

Accuracy: 42. Result ID: 69b90bf447f34ce924a16ba5.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-13B-Chat"}. This record does not assert independent reproduction.

### MoNE

Accuracy: 41.6. Result ID: 699e19595698665ec4d23e55.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

Recorded evaluation configuration: {"Pruning ratio":"50%","Base Model":"Qwen3-30B-A3B","Evaluation protocol":"Zero-shot","Calibration samples":100,"Calibration dataset":"Zyda2"}. This record does not assert independent reproduction.

### Ettin-Dec-1B

Accuracy: 41.6. Result ID: 69b3cdf447f34ce9249df1f4.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7a111e209131cea80ee7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7a111e209131cea80ee7.md>)

Recorded evaluation configuration: {"Model Size Category":"XL","Parameters":"908M-1.2B","Zero-shot":"true","Closed-book":"true"}. This record does not assert independent reproduction.

### Attacked

Accuracy: 41.4. Result ID: 69b90bf447f34ce924a16b99.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-7B-Chat"}. This record does not assert independent reproduction.

### Taylor

Accuracy: 41.4. Result ID: 6a06cd3247f34ce924ee15a9.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

Recorded evaluation configuration: {"Pruning Method":"Taylor","Base Model":"LLaMA3-8B","Calibration Dataset Size":"1,500-instance"}. This record does not assert independent reproduction.

### Out. Norm-Sim

Accuracy: 41.4. Result ID: 6a06cd3247f34ce924ee15ac.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75521e209131cea6727b.md>)

Recorded evaluation configuration: {"Pruning Method":"Out. Norm-Sim","Base Model":"LLaMA3-8B","Calibration Dataset Size":"1,500-instance"}. This record does not assert independent reproduction.

### Taylor

Accuracy: 41.4. Result ID: 6a06cd3247f34ce924ee15f5.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA3-8B","Pruning Ratio":"25%"}. This record does not assert independent reproduction.

### FT

Accuracy: 41.2. Result ID: 69b90bf447f34ce924a16ba0.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa70d01e209131cea4d60a.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-7B-Chat","Parameter Setting":"LoRA Adapter"}. This record does not assert independent reproduction.

### MoNE

Accuracy: 40.8. Result ID: 699e19595698665ec4d23e43.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

Recorded evaluation configuration: {"Pruning ratio":"50%","Base Model":"Moonlight","Evaluation protocol":"Zero-shot","Calibration samples":100,"Calibration dataset":"Zyda2"}. This record does not assert independent reproduction.

### RS

Accuracy: 40.8. Result ID: 699e19595698665ec4d23e48.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa777b1e209131cea7376a.md>)

Recorded evaluation configuration: {"Pruning ratio":"50%","Base Model":"Deepseek-V2-Lite","Evaluation protocol":"Zero-shot","Calibration samples":100,"Calibration dataset":"Zyda2"}. This record does not assert independent reproduction.

### BD-VAX

Accuracy: 40.6. Result ID: 69b90bf447f34ce924a16b9f.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-7B-Chat","Parameter Setting":"Full Params"}. This record does not assert independent reproduction.

### CROW

Accuracy: 40.2. Result ID: 69b90bf447f34ce924a16b9d.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa795d1e209131cea7d47c.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-7B-Chat","Parameter Setting":"Full Params"}. This record does not assert independent reproduction.

### Out. Cosine-Sim

Accuracy: 40.2. Result ID: 6a06cd3247f34ce924ee15ab.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75521e209131cea6727b.md>)

Recorded evaluation configuration: {"Pruning Method":"Out. Cosine-Sim","Base Model":"LLaMA3-8B","Calibration Dataset Size":"1,500-instance"}. This record does not assert independent reproduction.

### Pruning

Accuracy: 40. Result ID: 69b90bf447f34ce924a16b9b.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6b181e209131cea2c1a4.md>)

[Implementation](<https://github.com/csdongxian/ANP_backdoor>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-7B-Chat","Parameter Setting":"Full Params"}. This record does not assert independent reproduction.

### HyWIA

Accuracy: 39.6. Result ID: 69b272446dbb7075b95cd925.

[Reporting paper](<https://www.sota2.com/research/paper/68fa76891e209131cea6d801.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa76891e209131cea6d801.md>)

Recorded evaluation configuration: {"Ratio":"25%","Zero-shot":"true"}. This record does not assert independent reproduction.

### Out. Divergence-Sim

Accuracy: 39.6. Result ID: 6a06cd3247f34ce924ee15ad.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75521e209131cea6727b.md>)

Recorded evaluation configuration: {"Pruning Method":"Out. Divergence-Sim","Base Model":"LLaMA3-8B","Calibration Dataset Size":"1,500-instance"}. This record does not assert independent reproduction.

### Out. Norm-Sim

Accuracy: 39.6. Result ID: 6a06cd3247f34ce924ee15f8.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75521e209131cea6727b.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA3-8B","Pruning Ratio":"25%"}. This record does not assert independent reproduction.

### SALM-Duplex

Accuracy: 39.6. Result ID: 6a4f4b0360b88bb339e4bae6.

[Reporting paper](<https://www.sota2.com/research/paper/6a4f2b2b32fb937314875d39.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7a361e209131cea81d5a.md>)

Recorded evaluation configuration: {"Full Duplex":"true"}. This record does not assert independent reproduction.

### MC-SMoE

Accuracy: 39.4. Result ID: 699e19595698665ec4d23e53.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72841e209131cea56f90.md>)

Recorded evaluation configuration: {"Pruning ratio":"50%","Base Model":"Qwen3-30B-A3B","Evaluation protocol":"Zero-shot","Calibration samples":100,"Calibration dataset":"Zyda2"}. This record does not assert independent reproduction.

### Quantization

Accuracy: 39.4. Result ID: 69b90bf447f34ce924a16b9c.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa67901e209131cea170e6.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-7B-Chat","Parameter Setting":"Full Params"}. This record does not assert independent reproduction.

### Perplexity

Accuracy: 39.2. Result ID: 6a06cd3247f34ce924ee15ae.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

Recorded evaluation configuration: {"Pruning Method":"Perplexity","Base Model":"LLaMA3-8B","Calibration Dataset Size":"1,500-instance"}. This record does not assert independent reproduction.

### MC-SMoE

Accuracy: 38.8. Result ID: 699e19595698665ec4d23e4d.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72841e209131cea56f90.md>)

Recorded evaluation configuration: {"Pruning ratio":"50%","Base Model":"Qwen2-57B-A14B","Evaluation protocol":"Zero-shot","Calibration samples":100,"Calibration dataset":"Zyda2"}. This record does not assert independent reproduction.

### LLM-Pruner Vector⋆

Accuracy: 38.4. Result ID: 69b272446dbb7075b95cd923.

[Reporting paper](<https://www.sota2.com/research/paper/68fa76891e209131cea6d801.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa70af1e209131cea4c8b2.md>)

[Implementation](<https://github.com/horseee/LLM-Pruner>)

Recorded evaluation configuration: {"Ratio":"25%","Zero-shot":"true"}. This record does not assert independent reproduction.

### FLAP

Accuracy: 38.2. Result ID: 699e19595698665ec4d23e4c.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa718e1e209131cea51fae.md>)

Recorded evaluation configuration: {"Pruning ratio":"50%","Base Model":"Qwen2-57B-A14B","Evaluation protocol":"Zero-shot","Calibration samples":100,"Calibration dataset":"Zyda2"}. This record does not assert independent reproduction.

### RS

Accuracy: 38. Result ID: 699e19595698665ec4d23e42.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa777b1e209131cea7376a.md>)

Recorded evaluation configuration: {"Pruning ratio":"50%","Base Model":"Moonlight","Evaluation protocol":"Zero-shot","Calibration samples":100,"Calibration dataset":"Zyda2"}. This record does not assert independent reproduction.

### LLM-Pruner Element2⋆

Accuracy: 37.8. Result ID: 69b272446dbb7075b95cd924.

[Reporting paper](<https://www.sota2.com/research/paper/68fa76891e209131cea6d801.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa70af1e209131cea4c8b2.md>)

[Implementation](<https://github.com/horseee/LLM-Pruner>)

Recorded evaluation configuration: {"Ratio":"25%","Zero-shot":"true"}. This record does not assert independent reproduction.

### SmolLM2-360m

Accuracy: 37.6. Result ID: 69b3cdf447f34ce9249df1f0.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7a111e209131cea80ee7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa76d11e209131cea6f3fe.md>)

Recorded evaluation configuration: {"Model Size Category":"Large","Parameters":"360-410M","Zero-shot":"true","Closed-book":"true"}. This record does not assert independent reproduction.

### Pruning

Accuracy: 37.4. Result ID: 69b90bf447f34ce924a16ba7.

[Reporting paper](<https://www.sota2.com/research/paper/69b8e00c0636e5bf4afbea41.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6b181e209131cea2c1a4.md>)

[Implementation](<https://github.com/csdongxian/ANP_backdoor>)

Recorded evaluation configuration: {"Backbone":"LLaMA2-13B-Chat","Parameter Setting":"Full Params"}. This record does not assert independent reproduction.

### Llama-3.2-1B

Accuracy: 37.2. Result ID: 69b3cdf447f34ce9249df1f3.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7a111e209131cea80ee7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75d11e209131cea6a028.md>)

[Implementation](<https://github.com/meta-llama/llama-agentic-system>)

Recorded evaluation configuration: {"Model Size Category":"XL","Parameters":"908M-1.2B","Zero-shot":"true","Closed-book":"true"}. This record does not assert independent reproduction.

### MoNE

Accuracy: 36.8. Result ID: 699e19595698665ec4d23e49.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79ce1e209131cea7f78a.md>)

Recorded evaluation configuration: {"Pruning ratio":"50%","Base Model":"Deepseek-V2-Lite","Evaluation protocol":"Zero-shot","Calibration samples":100,"Calibration dataset":"Zyda2"}. This record does not assert independent reproduction.

### Cosine Similarity

Accuracy: 36.8. Result ID: 6a06cd3247f34ce924ee15aa.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75401e209131cea66d5a.md>)

Recorded evaluation configuration: {"Pruning Method":"Cosine Similarity","Base Model":"LLaMA3-8B","Calibration Dataset Size":"1,500-instance"}. This record does not assert independent reproduction.

### Slice-GPT

Accuracy: 36.8. Result ID: 6a06cd3247f34ce924ee15fb.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72271e209131cea54d13.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA3-8B","Pruning Ratio":"25%"}. This record does not assert independent reproduction.

### Cosine Similarity

Accuracy: 36.2. Result ID: 6a06cd3247f34ce924ee15f6.

[Reporting paper](<https://www.sota2.com/research/paper/6a06a89adbc836b87d82a06a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75401e209131cea66d5a.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA3-8B","Pruning Ratio":"25%"}. This record does not assert independent reproduction.

Page 1 of 2. This is an index page, not the full collection.

- [Next index page](<https://www.sota2.com/research/sota/question-answering-on-openbookqa-accuracy-and-performance-gain.md?metric=Accuracy&page=2>): Follow this link to discover more pages.
