# Natural Language Inference on MNLI (test) (Communication Overhead)

Source: [Natural Language Inference on MNLI (test) (Communication Overhead)](<https://www.sota2.com/research/sota/natural-language-inference-on-mnli-test-communication-overhead>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T06:07:31.001Z

Evaluation of Natural Language Inference capabilities on the MNLI test set, focusing specifically on communication efficiency metrics such as transmitted messages and overhead.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Natural Language Inference |
| Dataset (including any stated split) | MNLI (test) |
| Metric | Total Transmitted Messages (KB) |
| Unit | SIZE |
| Direction | Lower is better |
| Results loaded | 6 of 6 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Total Transmitted Messages (KB)](<https://www.sota2.com/research/sota/natural-language-inference-on-mnli-test-communication-overhead.md?metric=Total%20Transmitted%20Messages%20(KB)>): Lower is better; unit: SIZE.

- [Communication Overhead Multiplier](<https://www.sota2.com/research/sota/natural-language-inference-on-mnli-test-communication-overhead.md?metric=Communication%20Overhead%20Multiplier>): Lower is better; unit: SCALAR.

## Recorded results

| Method | Total Transmitted Messages (KB) | Result date | Report paper |
| --- | --- | --- | --- |
| FedTT+ | 78 | Not recorded | 68fa79061e209131cea7bc39 |
| FedTT | 234 | Not recorded | 68fa79061e209131cea7bc39 |
| BitFit | 390 | Not recorded | 68fa79061e209131cea7bc39 |
| ReLoRA | 624 | Not recorded | 68fa79061e209131cea7bc39 |
| LoRA | 1172 | Not recorded | 68fa79061e209131cea7bc39 |
| Prompt | 1523 | Not recorded | 68fa79061e209131cea7bc39 |

### FedTT+

Total Transmitted Messages (KB): 78. Result ID: 6995ba373badfaffe6bb865e.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79061e209131cea7bc39.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79061e209131cea7bc39.md>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### FedTT

Total Transmitted Messages (KB): 234. Result ID: 6995ba373badfaffe6bb865d.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79061e209131cea7bc39.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79061e209131cea7bc39.md>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### BitFit

Total Transmitted Messages (KB): 390. Result ID: 6995ba373badfaffe6bb865a.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79061e209131cea7bc39.md>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### ReLoRA

Total Transmitted Messages (KB): 624. Result ID: 6995ba373badfaffe6bb865b.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79061e209131cea7bc39.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa74ac1e209131cea631dd.md>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### LoRA

Total Transmitted Messages (KB): 1172. Result ID: 6995ba373badfaffe6bb8659.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79061e209131cea7bc39.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa728f1e209131cea57138.md>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### Prompt

Total Transmitted Messages (KB): 1523. Result ID: 6995ba373badfaffe6bb865c.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79061e209131cea7bc39.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6aa51e209131cea296c9.md>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.