# Natural Language Inference on FOLIO Source: [Natural Language Inference on FOLIO]() Collection index: [llms.txt]() Data updated: 2026-02-26T05:32:51.318Z Evaluation of Natural Language Inference performance on the FOLIO dataset, measured by Accuracy. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Natural Language Inference | | Dataset (including any stated split) | FOLIO | | Metric | Accuracy | | Unit | SCALAR | | Direction | Higher is better | | Results loaded | 26 of 26 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Accuracy](): Higher is better; unit: SCALAR. ## Recorded results | Method | Accuracy | Result date | Report paper | | --- | --- | --- | --- | | gemma-2-27b | 0.61 | Not recorded | 6993f6239ebc499c97409b3a | | gemma-2-9b | 0.53 | Not recorded | 6993f6239ebc499c97409b3a | | gemma-2-27b | 0.4 | Not recorded | 6993f6239ebc499c97409b3a | | gemma-2-9b | 0.38 | Not recorded | 6993f6239ebc499c97409b3a | | llama-3.2-1b | 0.38 | Not recorded | 6993f6239ebc499c97409b3a | | mistral-22b | 0.38 | Not recorded | 6993f6239ebc499c97409b3a | | llama-3.1-8b | 0.37 | Not recorded | 6993f6239ebc499c97409b3a | | ministral-8b | 0.37 | Not recorded | 6993f6239ebc499c97409b3a | | qwen-2.5-7b | 0.37 | Not recorded | 6993f6239ebc499c97409b3a | | qwen-2.5-14b | 0.37 | Not recorded | 6993f6239ebc499c97409b3a | | llama-3.2-3b | 0.36 | Not recorded | 6993f6239ebc499c97409b3a | | gemma-2-2b | 0.35 | Not recorded | 6993f6239ebc499c97409b3a | | qwen-2.5-1.5b | 0.34 | Not recorded | 6993f6239ebc499c97409b3a | | qwen-2.5-14b | 0.34 | Not recorded | 6993f6239ebc499c97409b3a | | qwen-2.5-3b | 0.32 | Not recorded | 6993f6239ebc499c97409b3a | | gemma-2-2b | 0.27 | Not recorded | 6993f6239ebc499c97409b3a | | qwen-2.5-0.5b | 0.26 | Not recorded | 6993f6239ebc499c97409b3a | | mistral-22b | 0.23 | Not recorded | 6993f6239ebc499c97409b3a | | qwen-2.5-1.5b | 0.19 | Not recorded | 6993f6239ebc499c97409b3a | | llama-3.2-3b | 0.18 | Not recorded | 6993f6239ebc499c97409b3a | | llama-3.2-1b | 0.17 | Not recorded | 6993f6239ebc499c97409b3a | | qwen-2.5-0.5b | 0.17 | Not recorded | 6993f6239ebc499c97409b3a | | ministral-8b | 0.16 | Not recorded | 6993f6239ebc499c97409b3a | | llama-3.1-8b | 0.12 | Not recorded | 6993f6239ebc499c97409b3a | | qwen-2.5-3b | 0.08 | Not recorded | 6993f6239ebc499c97409b3a | | qwen-2.5-7b | 0.01 | Not recorded | 6993f6239ebc499c97409b3a | ### gemma-2-27b Accuracy: 0.61. Result ID: 699a70b15698665ec4f9772f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"GCD"}. This record does not assert independent reproduction. ### gemma-2-9b Accuracy: 0.53. Result ID: 699a70b15698665ec4f9772d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"GCD"}. This record does not assert independent reproduction. ### gemma-2-27b Accuracy: 0.4. Result ID: 699a70b15698665ec4f97730. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"NL2LOGIC"}. This record does not assert independent reproduction. ### gemma-2-9b Accuracy: 0.38. Result ID: 699a70b15698665ec4f9772e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"NL2LOGIC"}. This record does not assert independent reproduction. ### llama-3.2-1b Accuracy: 0.38. Result ID: 699a70b15698665ec4f97732. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"NL2LOGIC"}. This record does not assert independent reproduction. ### mistral-22b Accuracy: 0.38. Result ID: 699a70b15698665ec4f9773a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"NL2LOGIC"}. This record does not assert independent reproduction. ### llama-3.1-8b Accuracy: 0.37. Result ID: 699a70b15698665ec4f97736. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"NL2LOGIC"}. This record does not assert independent reproduction. ### ministral-8b Accuracy: 0.37. Result ID: 699a70b15698665ec4f97738. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"NL2LOGIC"}. This record does not assert independent reproduction. ### qwen-2.5-7b Accuracy: 0.37. Result ID: 699a70b15698665ec4f97742. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"NL2LOGIC"}. This record does not assert independent reproduction. ### qwen-2.5-14b Accuracy: 0.37. Result ID: 699a70b15698665ec4f97744. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"NL2LOGIC"}. This record does not assert independent reproduction. ### llama-3.2-3b Accuracy: 0.36. Result ID: 699a70b15698665ec4f97734. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"NL2LOGIC"}. This record does not assert independent reproduction. ### gemma-2-2b Accuracy: 0.35. Result ID: 699a70b15698665ec4f9772c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"NL2LOGIC"}. This record does not assert independent reproduction. ### qwen-2.5-1.5b Accuracy: 0.34. Result ID: 699a70b15698665ec4f9773e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"NL2LOGIC"}. This record does not assert independent reproduction. ### qwen-2.5-14b Accuracy: 0.34. Result ID: 699a70b15698665ec4f97743. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"GCD"}. This record does not assert independent reproduction. ### qwen-2.5-3b Accuracy: 0.32. Result ID: 699a70b15698665ec4f97740. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"NL2LOGIC"}. This record does not assert independent reproduction. ### gemma-2-2b Accuracy: 0.27. Result ID: 699a70b15698665ec4f9772b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"GCD"}. This record does not assert independent reproduction. ### qwen-2.5-0.5b Accuracy: 0.26. Result ID: 699a70b15698665ec4f9773c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"NL2LOGIC"}. This record does not assert independent reproduction. ### mistral-22b Accuracy: 0.23. Result ID: 699a70b15698665ec4f97739. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"GCD"}. This record does not assert independent reproduction. ### qwen-2.5-1.5b Accuracy: 0.19. Result ID: 699a70b15698665ec4f9773d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"GCD"}. This record does not assert independent reproduction. ### llama-3.2-3b Accuracy: 0.18. Result ID: 699a70b15698665ec4f97733. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"GCD"}. This record does not assert independent reproduction. ### llama-3.2-1b Accuracy: 0.17. Result ID: 699a70b15698665ec4f97731. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"GCD"}. This record does not assert independent reproduction. ### qwen-2.5-0.5b Accuracy: 0.17. Result ID: 699a70b15698665ec4f9773b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"GCD"}. This record does not assert independent reproduction. ### ministral-8b Accuracy: 0.16. Result ID: 699a70b15698665ec4f97737. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"GCD"}. This record does not assert independent reproduction. ### llama-3.1-8b Accuracy: 0.12. Result ID: 699a70b15698665ec4f97735. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"GCD"}. This record does not assert independent reproduction. ### qwen-2.5-3b Accuracy: 0.08. Result ID: 699a70b15698665ec4f9773f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"GCD"}. This record does not assert independent reproduction. ### qwen-2.5-7b Accuracy: 0.01. Result ID: 699a70b15698665ec4f97741. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Decoding Method":"GCD"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.