# Question Answering on ARC Challenge 25-shot Source: [Question Answering on ARC Challenge 25-shot]() Collection index: [llms.txt]() Data updated: 2026-04-28T09:32:45.850Z Evaluation of model performance on the 25-shot setting of the ARC Challenge dataset for Question Answering, measured by Accuracy. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Question Answering | | Dataset (including any stated split) | ARC Challenge 25-shot | | Metric | Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 18 of 18 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Accuracy](): Higher is better; unit: PERCENT. ## Recorded results | Method | Accuracy | Result date | Report paper | | --- | --- | --- | --- | | FP16 | 61.2 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | COVERCAL | 60.7 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | Max-ActVar | 59.8 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | Stratified | 59.7 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | Random | 59.5 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | FP16 | 59.4 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | Max-PPL | 59.3 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | COVERCAL | 58.7 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | Max-ActVar | 57.6 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | Stratified | 57.4 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | Random | 57.2 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | Max-PPL | 57 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | FP16 | 53.1 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | COVERCAL | 52.5 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | Max-ActVar | 51.6 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | Stratified | 51.5 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | Random | 51.3 | Not recorded | 69f03f2d55a3c32af89fa1b3 | | Max-PPL | 51 | Not recorded | 69f03f2d55a3c32af89fa1b3 | ### FP16 Accuracy: 61.2. Result ID: 69f07eac47f34ce924d6c382. [Reporting paper]() Recorded evaluation configuration: {"Model":"Mistral-7B","Quantization":"FP16"}. This record does not assert independent reproduction. ### COVERCAL Accuracy: 60.7. Result ID: 69f07eac47f34ce924d6c387. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Mistral-7B","Quantization":"AWQ INT4","K (Calibration Samples)":"128"}. This record does not assert independent reproduction. ### Max-ActVar Accuracy: 59.8. Result ID: 69f07eac47f34ce924d6c385. [Reporting paper]() Recorded evaluation configuration: {"Model":"Mistral-7B","Quantization":"AWQ INT4","K (Calibration Samples)":"128"}. This record does not assert independent reproduction. ### Stratified Accuracy: 59.7. Result ID: 69f07eac47f34ce924d6c386. [Reporting paper]() Recorded evaluation configuration: {"Model":"Mistral-7B","Quantization":"AWQ INT4","K (Calibration Samples)":"128"}. This record does not assert independent reproduction. ### Random Accuracy: 59.5. Result ID: 69f07eac47f34ce924d6c383. [Reporting paper]() Recorded evaluation configuration: {"Model":"Mistral-7B","Quantization":"AWQ INT4","K (Calibration Samples)":"128"}. This record does not assert independent reproduction. ### FP16 Accuracy: 59.4. Result ID: 69f07eac47f34ce924d6c37c. [Reporting paper]() Recorded evaluation configuration: {"Model":"LLaMA-3-8B","Quantization":"FP16"}. This record does not assert independent reproduction. ### Max-PPL Accuracy: 59.3. Result ID: 69f07eac47f34ce924d6c384. [Reporting paper]() Recorded evaluation configuration: {"Model":"Mistral-7B","Quantization":"AWQ INT4","K (Calibration Samples)":"128"}. This record does not assert independent reproduction. ### COVERCAL Accuracy: 58.7. Result ID: 69f07eac47f34ce924d6c381. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"LLaMA-3-8B","Quantization":"AWQ INT4","K (Calibration Samples)":"128"}. This record does not assert independent reproduction. ### Max-ActVar Accuracy: 57.6. Result ID: 69f07eac47f34ce924d6c37f. [Reporting paper]() Recorded evaluation configuration: {"Model":"LLaMA-3-8B","Quantization":"AWQ INT4","K (Calibration Samples)":"128"}. This record does not assert independent reproduction. ### Stratified Accuracy: 57.4. Result ID: 69f07eac47f34ce924d6c380. [Reporting paper]() Recorded evaluation configuration: {"Model":"LLaMA-3-8B","Quantization":"AWQ INT4","K (Calibration Samples)":"128"}. This record does not assert independent reproduction. ### Random Accuracy: 57.2. Result ID: 69f07eac47f34ce924d6c37d. [Reporting paper]() Recorded evaluation configuration: {"Model":"LLaMA-3-8B","Quantization":"AWQ INT4","K (Calibration Samples)":"128"}. This record does not assert independent reproduction. ### Max-PPL Accuracy: 57. Result ID: 69f07eac47f34ce924d6c37e. [Reporting paper]() Recorded evaluation configuration: {"Model":"LLaMA-3-8B","Quantization":"AWQ INT4","K (Calibration Samples)":"128"}. This record does not assert independent reproduction. ### FP16 Accuracy: 53.1. Result ID: 69f07eac47f34ce924d6c376. [Reporting paper]() Recorded evaluation configuration: {"Model":"LLaMA-2-7B","Quantization":"FP16"}. This record does not assert independent reproduction. ### COVERCAL Accuracy: 52.5. Result ID: 69f07eac47f34ce924d6c37b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"LLaMA-2-7B","Quantization":"AWQ INT4","K (Calibration Samples)":"128"}. This record does not assert independent reproduction. ### Max-ActVar Accuracy: 51.6. Result ID: 69f07eac47f34ce924d6c379. [Reporting paper]() Recorded evaluation configuration: {"Model":"LLaMA-2-7B","Quantization":"AWQ INT4","K (Calibration Samples)":"128"}. This record does not assert independent reproduction. ### Stratified Accuracy: 51.5. Result ID: 69f07eac47f34ce924d6c37a. [Reporting paper]() Recorded evaluation configuration: {"Model":"LLaMA-2-7B","Quantization":"AWQ INT4","K (Calibration Samples)":"128"}. This record does not assert independent reproduction. ### Random Accuracy: 51.3. Result ID: 69f07eac47f34ce924d6c377. [Reporting paper]() Recorded evaluation configuration: {"Model":"LLaMA-2-7B","Quantization":"AWQ INT4","K (Calibration Samples)":"128"}. This record does not assert independent reproduction. ### Max-PPL Accuracy: 51. Result ID: 69f07eac47f34ce924d6c378. [Reporting paper]() Recorded evaluation configuration: {"Model":"LLaMA-2-7B","Quantization":"AWQ INT4","K (Calibration Samples)":"128"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.