# Large Language Model Evaluation on AlpacaEval, TruthfulQA, and MMLU (test)

Source: [Large Language Model Evaluation on AlpacaEval, TruthfulQA, and MMLU (test)](<https://www.sota2.com/research/sota/large-language-model-evaluation-on-alpacaeval-truthfulqa-and-mmlu-test>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-18T08:12:11.601Z

Evaluation of Large Language Model performance specifically on the SFT Evaluation Suite components: AlpacaEval, TruthfulQA, and MMLU, using the test split.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Large Language Model Evaluation |
| Dataset (including any stated split) | SFT Evaluation Suite (AlpacaEval, TruthfulQA, MMLU) (test) |
| Metric | AlpacaEval Score |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 7 of 7 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [AlpacaEval Score](<https://www.sota2.com/research/sota/large-language-model-evaluation-on-alpacaeval-truthfulqa-and-mmlu-test.md?metric=AlpacaEval%20Score>): Higher is better; unit: PERCENT.

- [TruthfulQA Score](<https://www.sota2.com/research/sota/large-language-model-evaluation-on-alpacaeval-truthfulqa-and-mmlu-test.md?metric=TruthfulQA%20Score>): Higher is better; unit: PERCENT.

- [MMLU Score](<https://www.sota2.com/research/sota/large-language-model-evaluation-on-alpacaeval-truthfulqa-and-mmlu-test.md?metric=MMLU%20Score>): Higher is better; unit: PERCENT.

- [Average Score](<https://www.sota2.com/research/sota/large-language-model-evaluation-on-alpacaeval-truthfulqa-and-mmlu-test.md?metric=Average%20Score>): Higher is better; unit: PERCENT.

## Recorded results

| Method | AlpacaEval Score | Result date | Report paper |
| --- | --- | --- | --- |
| Warmup-Stable-Only (WSO) | 78.1 | Not recorded | 69ba31ab0636e5bf4afc09be |
| WSD | 77.2 | Not recorded | 69ba31ab0636e5bf4afc09be |
| Cosine | 76.4 | Not recorded | 69ba31ab0636e5bf4afc09be |
| WSD | 76 | Not recorded | 69ba31ab0636e5bf4afc09be |
| Cosine | 76 | Not recorded | 69ba31ab0636e5bf4afc09be |
| Linear | 75.6 | Not recorded | 69ba31ab0636e5bf4afc09be |
| Linear | 75.5 | Not recorded | 69ba31ab0636e5bf4afc09be |

### Warmup-Stable-Only (WSO)

AlpacaEval Score: 78.1. Result ID: 69ba5e5b47f34ce924a41f7f.

[Reporting paper](<https://www.sota2.com/research/paper/69ba31ab0636e5bf4afc09be.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69ba31ab0636e5bf4afc09be.md>)

Recorded evaluation configuration: {"Model":"1B","Scheduler":"Warmup-Stable-Only (WSO)","alpha\_pre":"1.0","Pre-training tokens":"2T","Selected SFT LR":"1 × 10−4"}. This record does not assert independent reproduction.

### WSD

AlpacaEval Score: 77.2. Result ID: 69ba5e5b47f34ce924a41f80.

[Reporting paper](<https://www.sota2.com/research/paper/69ba31ab0636e5bf4afc09be.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73671e209131cea5c3ab.md>)

Recorded evaluation configuration: {"Model":"1B","Scheduler":"WSD","alpha\_pre":"0.1","Pre-training tokens":"2T","Selected SFT LR":"3 × 10−5"}. This record does not assert independent reproduction.

### Cosine

AlpacaEval Score: 76.4. Result ID: 69ba5e5b47f34ce924a41f85.

[Reporting paper](<https://www.sota2.com/research/paper/69ba31ab0636e5bf4afc09be.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa680c1e209131cea19d53.md>)

[Implementation](<https://github.com/openai/gpt-3/blob/master/overlap_frequency.md>)

Recorded evaluation configuration: {"Model":"1B","Scheduler":"Cosine","alpha\_pre":"0.0","Pre-training tokens":"2T","Selected SFT LR":"1 × 10−4"}. This record does not assert independent reproduction.

### WSD

AlpacaEval Score: 76. Result ID: 69ba5e5b47f34ce924a41f81.

[Reporting paper](<https://www.sota2.com/research/paper/69ba31ab0636e5bf4afc09be.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73671e209131cea5c3ab.md>)

Recorded evaluation configuration: {"Model":"1B","Scheduler":"WSD","alpha\_pre":"0.0","Pre-training tokens":"2T","Selected SFT LR":"3 × 10−5"}. This record does not assert independent reproduction.

### Cosine

AlpacaEval Score: 76. Result ID: 69ba5e5b47f34ce924a41f84.

[Reporting paper](<https://www.sota2.com/research/paper/69ba31ab0636e5bf4afc09be.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa680c1e209131cea19d53.md>)

[Implementation](<https://github.com/openai/gpt-3/blob/master/overlap_frequency.md>)

Recorded evaluation configuration: {"Model":"1B","Scheduler":"Cosine","alpha\_pre":"0.1","Pre-training tokens":"2T","Selected SFT LR":"1 × 10−5"}. This record does not assert independent reproduction.

### Linear

AlpacaEval Score: 75.6. Result ID: 69ba5e5b47f34ce924a41f82.

[Reporting paper](<https://www.sota2.com/research/paper/69ba31ab0636e5bf4afc09be.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa772c1e209131cea71903.md>)

Recorded evaluation configuration: {"Model":"1B","Scheduler":"Linear","alpha\_pre":"0.1","Pre-training tokens":"2T","Selected SFT LR":"3 × 10−5"}. This record does not assert independent reproduction.

### Linear

AlpacaEval Score: 75.5. Result ID: 69ba5e5b47f34ce924a41f83.

[Reporting paper](<https://www.sota2.com/research/paper/69ba31ab0636e5bf4afc09be.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa772c1e209131cea71903.md>)

Recorded evaluation configuration: {"Model":"1B","Scheduler":"Linear","alpha\_pre":"0.0","Pre-training tokens":"2T","Selected SFT LR":"3 × 10−5"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.