# Downstream NLP Evaluation on LM Evaluation Harness (test) Source: [Downstream NLP Evaluation on LM Evaluation Harness (test)]() Collection index: [llms.txt]() Data updated: 2026-02-26T05:48:24.123Z Evaluation of model performance across a suite of nine NLP tasks (Lambada, ARC, WinoGrande, PIQA, HellaSwag, SciQ, RACE) using the LM Evaluation Harness test configuration. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Downstream NLP Evaluation | | Dataset (including any stated split) | Downstream NLP Tasks (Lambada, ARC, WinoGrande, PIQA, HellaSwag, SciQ, RACE) LM Evaluation Harness (test) | | Metric | Lambada OpenAI | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 36 of 36 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Lambada OpenAI](): Higher is better; unit: PERCENT. - [ARC-E](): Higher is better; unit: PERCENT. - [Lambada Standard](): Higher is better; unit: PERCENT. - [ARC-C](): Higher is better; unit: PERCENT. - [WinoGrande](): Higher is better; unit: PERCENT. - [PIQA](): Higher is better; unit: PERCENT. - [HellaSwag](): Higher is better; unit: PERCENT. - [SciQ](): Higher is better; unit: PERCENT. - [RACE](): Higher is better; unit: PERCENT. - [Average Accuracy](): Higher is better; unit: PERCENT. - [Accuracy Delta](): Higher is better; unit: SCALAR. ## Recorded results | Method | Lambada OpenAI | Result date | Report paper | | --- | --- | --- | --- | | Pythia-12B | 70.4 | Not recorded | 68fa79a91e209131cea7ebef | | PonderPythia-2.8B | 68.9 | Not recorded | 68fa79a91e209131cea7ebef | | Pythia-6.9B | 67.2 | Not recorded | 68fa79a91e209131cea7ebef | | Pythia-12B | 66.5 | Not recorded | 68fa79a91e209131cea7ebef | | PonderPythia-1.4B | 65.2 | Not recorded | 68fa79a91e209131cea7ebef | | Pythia-2.8B | 64.6 | Not recorded | 68fa79a91e209131cea7ebef | | PonderPythia-2.8B | 64.2 | Not recorded | 68fa79a91e209131cea7ebef | | OPT-2.7B | 63.5 | Not recorded | 68fa79a91e209131cea7ebef | | Pythia-6.9B | 62.5 | Not recorded | 68fa79a91e209131cea7ebef | | PonderPythia-1B | 62.3 | Not recorded | 68fa79a91e209131cea7ebef | | Pythia-1.4B | 61.6 | Not recorded | 68fa79a91e209131cea7ebef | | OPT-2.7B | 60.2 | Not recorded | 68fa79a91e209131cea7ebef | | PonderPythia-1.4B | 59.2 | Not recorded | 68fa79a91e209131cea7ebef | | Pythia-2.8B | 59 | Not recorded | 68fa79a91e209131cea7ebef | | Tinyllama-1.1B | 58.8 | Not recorded | 68fa79a91e209131cea7ebef | | OPT-1.3B | 57.9 | Not recorded | 68fa79a91e209131cea7ebef | | PonderPythia-1B | 57.7 | Not recorded | 68fa79a91e209131cea7ebef | | PonderPythia-410M | 56.9 | Not recorded | 68fa79a91e209131cea7ebef | | Pythia-1B | 55.9 | Not recorded | 68fa79a91e209131cea7ebef | | Pythia-1.4B | 54.5 | Not recorded | 68fa79a91e209131cea7ebef | | OPT-1.3B | 54 | Not recorded | 68fa79a91e209131cea7ebef | | Tinyllama-1.1B | 53.8 | Not recorded | 68fa79a91e209131cea7ebef | | Bloom-3B | 51.7 | Not recorded | 68fa79a91e209131cea7ebef | | Pythia-410M | 51.4 | Not recorded | 68fa79a91e209131cea7ebef | | PonderPythia-410M | 48.9 | Not recorded | 68fa79a91e209131cea7ebef | | Pythia-1B | 48.3 | Not recorded | 68fa79a91e209131cea7ebef | | Bloom-3B | 46.2 | Not recorded | 68fa79a91e209131cea7ebef | | Bloom-1.7B | 46.2 | Not recorded | 68fa79a91e209131cea7ebef | | OPT-350M | 45.2 | Not recorded | 68fa79a91e209131cea7ebef | | Pythia-410M | 43.9 | Not recorded | 68fa79a91e209131cea7ebef | | Bloom-1.1B | 42.6 | Not recorded | 68fa79a91e209131cea7ebef | | Bloom-1.7B | 42.5 | Not recorded | 68fa79a91e209131cea7ebef | | OPT-350M | 38.3 | Not recorded | 68fa79a91e209131cea7ebef | | Bloom-1.1B | 36.3 | Not recorded | 68fa79a91e209131cea7ebef | | Bloom-560M | 34.3 | Not recorded | 68fa79a91e209131cea7ebef | | Bloom-560M | 29.4 | Not recorded | 68fa79a91e209131cea7ebef | ### Pythia-12B Lambada OpenAI: 70.4. Result ID: 699cfef85698665ec49f1e60. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### PonderPythia-2.8B Lambada OpenAI: 68.9. Result ID: 699cfef85698665ec49f1e61. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### Pythia-6.9B Lambada OpenAI: 67.2. Result ID: 699cfef85698665ec49f1e5f. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### Pythia-12B Lambada OpenAI: 66.5. Result ID: 699cfef85698665ec49f1e4e. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### PonderPythia-1.4B Lambada OpenAI: 65.2. Result ID: 699cfef85698665ec49f1e5b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### Pythia-2.8B Lambada OpenAI: 64.6. Result ID: 699cfef85698665ec49f1e5c. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### PonderPythia-2.8B Lambada OpenAI: 64.2. Result ID: 699cfef85698665ec49f1e4f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### OPT-2.7B Lambada OpenAI: 63.5. Result ID: 699cfef85698665ec49f1e5d. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### Pythia-6.9B Lambada OpenAI: 62.5. Result ID: 699cfef85698665ec49f1e4d. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### PonderPythia-1B Lambada OpenAI: 62.3. Result ID: 699cfef85698665ec49f1e58. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### Pythia-1.4B Lambada OpenAI: 61.6. Result ID: 699cfef85698665ec49f1e59. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### OPT-2.7B Lambada OpenAI: 60.2. Result ID: 699cfef85698665ec49f1e4b. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### PonderPythia-1.4B Lambada OpenAI: 59.2. Result ID: 699cfef85698665ec49f1e49. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### Pythia-2.8B Lambada OpenAI: 59. Result ID: 699cfef85698665ec49f1e4a. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### Tinyllama-1.1B Lambada OpenAI: 58.8. Result ID: 699cfef85698665ec49f1e57. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"3T"}. This record does not assert independent reproduction. ### OPT-1.3B Lambada OpenAI: 57.9. Result ID: 699cfef85698665ec49f1e55. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### PonderPythia-1B Lambada OpenAI: 57.7. Result ID: 699cfef85698665ec49f1e46. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### PonderPythia-410M Lambada OpenAI: 56.9. Result ID: 699cfef85698665ec49f1e53. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### Pythia-1B Lambada OpenAI: 55.9. Result ID: 699cfef85698665ec49f1e54. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### Pythia-1.4B Lambada OpenAI: 54.5. Result ID: 699cfef85698665ec49f1e47. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### OPT-1.3B Lambada OpenAI: 54. Result ID: 699cfef85698665ec49f1e43. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### Tinyllama-1.1B Lambada OpenAI: 53.8. Result ID: 699cfef85698665ec49f1e45. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"3T"}. This record does not assert independent reproduction. ### Bloom-3B Lambada OpenAI: 51.7. Result ID: 699cfef85698665ec49f1e5e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"366B"}. This record does not assert independent reproduction. ### Pythia-410M Lambada OpenAI: 51.4. Result ID: 699cfef85698665ec49f1e50. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### PonderPythia-410M Lambada OpenAI: 48.9. Result ID: 699cfef85698665ec49f1e41. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### Pythia-1B Lambada OpenAI: 48.3. Result ID: 699cfef85698665ec49f1e42. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### Bloom-3B Lambada OpenAI: 46.2. Result ID: 699cfef85698665ec49f1e4c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"366B"}. This record does not assert independent reproduction. ### Bloom-1.7B Lambada OpenAI: 46.2. Result ID: 699cfef85698665ec49f1e5a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"366B"}. This record does not assert independent reproduction. ### OPT-350M Lambada OpenAI: 45.2. Result ID: 699cfef85698665ec49f1e51. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### Pythia-410M Lambada OpenAI: 43.9. Result ID: 699cfef85698665ec49f1e3e. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### Bloom-1.1B Lambada OpenAI: 42.6. Result ID: 699cfef85698665ec49f1e56. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"366B"}. This record does not assert independent reproduction. ### Bloom-1.7B Lambada OpenAI: 42.5. Result ID: 699cfef85698665ec49f1e48. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"366B"}. This record does not assert independent reproduction. ### OPT-350M Lambada OpenAI: 38.3. Result ID: 699cfef85698665ec49f1e3f. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"300B"}. This record does not assert independent reproduction. ### Bloom-1.1B Lambada OpenAI: 36.3. Result ID: 699cfef85698665ec49f1e44. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"366B"}. This record does not assert independent reproduction. ### Bloom-560M Lambada OpenAI: 34.3. Result ID: 699cfef85698665ec49f1e52. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"0-shot","Number of training tokens":"366B"}. This record does not assert independent reproduction. ### Bloom-560M Lambada OpenAI: 29.4. Result ID: 699cfef85698665ec49f1e40. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Shots":"5-shot","Number of training tokens":"366B"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.