# Character Recognition on FEMNIST (test)

Source: [Character Recognition on FEMNIST (test)](<https://www.sota2.com/research/sota/character-recognition-on-femnist-test>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-18T06:41:57.488Z

Evaluation of Character Recognition performance on the test split of the FEMNIST dataset, using specialized metrics often associated with fairness and robustness analysis in federated learning (e.g., SIA, EOD, Welfare).

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Character Recognition |
| Dataset (including any stated split) | FEMNIST (test) |
| Metric | Accuracy |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 7 of 7 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Accuracy](<https://www.sota2.com/research/sota/character-recognition-on-femnist-test.md?metric=Accuracy>): Higher is better; unit: PERCENT.

- [Mean SIA Accuracy](<https://www.sota2.com/research/sota/character-recognition-on-femnist-test.md?metric=Mean%20SIA%20Accuracy>): Higher is better; unit: PERCENT.

- [Max SIA Accuracy](<https://www.sota2.com/research/sota/character-recognition-on-femnist-test.md?metric=Max%20SIA%20Accuracy>): Higher is better; unit: PERCENT.

- [CoV SIA Accuracy](<https://www.sota2.com/research/sota/character-recognition-on-femnist-test.md?metric=CoV%20SIA%20Accuracy>): Lower is better; unit: PERCENT.

- [FI SIA Accuracy](<https://www.sota2.com/research/sota/character-recognition-on-femnist-test.md?metric=FI%20SIA%20Accuracy>): Higher is better; unit: PERCENT.

- [CoV Loss](<https://www.sota2.com/research/sota/character-recognition-on-femnist-test.md?metric=CoV%20Loss>): Lower is better; unit: PERCENT.

- [FI Loss](<https://www.sota2.com/research/sota/character-recognition-on-femnist-test.md?metric=FI%20Loss>): Higher is better; unit: PERCENT.

- [EOD](<https://www.sota2.com/research/sota/character-recognition-on-femnist-test.md?metric=EOD>): Lower is better; unit: PERCENT.

- [MAD Loss](<https://www.sota2.com/research/sota/character-recognition-on-femnist-test.md?metric=MAD%20Loss>): Lower is better; unit: SCALAR.

- [Welfare (SIA)](<https://www.sota2.com/research/sota/character-recognition-on-femnist-test.md?metric=Welfare%20(SIA)>): Higher is better; unit: PERCENT.

- [Communication Rounds](<https://www.sota2.com/research/sota/character-recognition-on-femnist-test.md?metric=Communication%20Rounds>): Lower is better; unit: SCALAR.

## Recorded results

| Method | Accuracy | Result date | Report paper |
| --- | --- | --- | --- |
| Baseline | 65.04 | Not recorded | 68fa7be31e209131cea8b0f6 |
| FinP | 64.23 | Not recorded | 68fa7be31e209131cea8b0f6 |
| FinP | 63.76 | Not recorded | 68fa7be31e209131cea8b0f6 |
| FinP | 60.11 | Not recorded | 68fa7be31e209131cea8b0f6 |
| DP-SGD | 49.97 | Not recorded | 68fa7be31e209131cea8b0f6 |
| DP-SGD | 29.41 | Not recorded | 68fa7be31e209131cea8b0f6 |
| DP-SGD | 5.68 | Not recorded | 68fa7be31e209131cea8b0f6 |

### Baseline

Accuracy: 65.04. Result ID: 6a33932efbc373592c3034e4.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7be31e209131cea8b0f6.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa70be1e209131cea4d02c.md>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### FinP

Accuracy: 64.23. Result ID: 6a33932efbc373592c3034e7.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7be31e209131cea8b0f6.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7be31e209131cea8b0f6.md>)

Recorded evaluation configuration: {"beta (β)":"0.5"}. This record does not assert independent reproduction.

### FinP

Accuracy: 63.76. Result ID: 6a33932efbc373592c3034e6.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7be31e209131cea8b0f6.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7be31e209131cea8b0f6.md>)

Recorded evaluation configuration: {"beta (β)":"0.75"}. This record does not assert independent reproduction.

### FinP

Accuracy: 60.11. Result ID: 6a33932efbc373592c3034e5.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7be31e209131cea8b0f6.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7be31e209131cea8b0f6.md>)

Recorded evaluation configuration: {"beta (β)":"1"}. This record does not assert independent reproduction.

### DP-SGD

Accuracy: 49.97. Result ID: 6a33932efbc373592c3034e8.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7be31e209131cea8b0f6.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa65301e209131cea08bf5.md>)

Recorded evaluation configuration: {"epsilon (ε)":"151","delta (δ)":"10^{-5}"}. This record does not assert independent reproduction.

### DP-SGD

Accuracy: 29.41. Result ID: 6a33932efbc373592c3034e9.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7be31e209131cea8b0f6.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa65301e209131cea08bf5.md>)

Recorded evaluation configuration: {"epsilon (ε)":"92","delta (δ)":"10^{-5}"}. This record does not assert independent reproduction.

### DP-SGD

Accuracy: 5.68. Result ID: 6a33932efbc373592c3034ea.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7be31e209131cea8b0f6.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa65301e209131cea08bf5.md>)

Recorded evaluation configuration: {"epsilon (ε)":"31","delta (δ)":"10^{-5}"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
