# Long-context Language Understanding on LongBench (7-Category Performance Summary)

Source: [Long-context Language Understanding on LongBench (7-Category Performance Summary)](<https://www.sota2.com/research/sota/long-context-language-understanding-on-longbench-7-category-performance-summary>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-19T08:11:32.255Z

Evaluation of LongBench focusing specifically on 7 core capability pillars: Mathematical reasoning, Code generation, Few-shot learning, Multi-QA, Single-QA, Summarization, and Synthetic task performance.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Long-context Language Understanding |
| Dataset (including any stated split) | LongBench |
| Metric | Math Performance |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 15 of 15 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Math Performance](<https://www.sota2.com/research/sota/long-context-language-understanding-on-longbench-7-category-performance-summary.md?metric=Math%20Performance>): Higher is better; unit: PERCENT.

- [Code Generation Performance](<https://www.sota2.com/research/sota/long-context-language-understanding-on-longbench-7-category-performance-summary.md?metric=Code%20Generation%20Performance>): Higher is better; unit: PERCENT.

- [Few-shot Learning Performance](<https://www.sota2.com/research/sota/long-context-language-understanding-on-longbench-7-category-performance-summary.md?metric=Few-shot%20Learning%20Performance>): Higher is better; unit: PERCENT.

- [Multi-QA Performance](<https://www.sota2.com/research/sota/long-context-language-understanding-on-longbench-7-category-performance-summary.md?metric=Multi-QA%20Performance>): Higher is better; unit: PERCENT.

- [Single-QA Performance](<https://www.sota2.com/research/sota/long-context-language-understanding-on-longbench-7-category-performance-summary.md?metric=Single-QA%20Performance>): Higher is better; unit: PERCENT.

- [Summarization Performance](<https://www.sota2.com/research/sota/long-context-language-understanding-on-longbench-7-category-performance-summary.md?metric=Summarization%20Performance>): Higher is better; unit: PERCENT.

- [Synthetic Task Performance](<https://www.sota2.com/research/sota/long-context-language-understanding-on-longbench-7-category-performance-summary.md?metric=Synthetic%20Task%20Performance>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Math Performance | Result date | Report paper |
| --- | --- | --- | --- |
| 16-bit Baseline | 80.29 | Not recorded | 68fa77a11e209131cea747e1 |
| 16-bit Baseline | 71.42 | Not recorded | 68fa77a11e209131cea747e1 |
| 16-bit Baseline | 70.28 | Not recorded | 68fa77a11e209131cea747e1 |
| LogQuant | 63.31 | Not recorded | 68fa77a11e209131cea747e1 |
| KiVi | 59.82 | Not recorded | 68fa77a11e209131cea747e1 |
| 16-bit Baseline | 56.18 | Not recorded | 68fa77a11e209131cea747e1 |
| 16-bit Baseline | 52.99 | Not recorded | 68fa77a11e209131cea747e1 |
| LogQuant | 51.86 | Not recorded | 68fa77a11e209131cea747e1 |
| LogQuant | 49.28 | Not recorded | 68fa77a11e209131cea747e1 |
| LogQuant | 40.41 | Not recorded | 68fa77a11e209131cea747e1 |
| KiVi | 39.27 | Not recorded | 68fa77a11e209131cea747e1 |
| LogQuant | 34.34 | Not recorded | 68fa77a11e209131cea747e1 |
| KiVi | 18.04 | Not recorded | 68fa77a11e209131cea747e1 |
| KiVi | 12.59 | Not recorded | 68fa77a11e209131cea747e1 |
| KiVi | 3.71 | Not recorded | 68fa77a11e209131cea747e1 |

### 16-bit Baseline

Math Performance: 80.29. Result ID: 6a0c1af0fbc373592c020fa6.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

Recorded evaluation configuration: {"Model":"Phi-3-mini-128k-instruct","KV Cache Precision":"16-bit"}. This record does not assert independent reproduction.

### 16-bit Baseline

Math Performance: 71.42. Result ID: 6a0c1af0fbc373592c020f9a.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

Recorded evaluation configuration: {"Model":"llama-3.1-8B-Instruct","KV Cache Precision":"16-bit"}. This record does not assert independent reproduction.

### 16-bit Baseline

Math Performance: 70.28. Result ID: 6a0c1af0fbc373592c020fa0.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

Recorded evaluation configuration: {"Model":"Qwen1.5-14B-Chat-AWQ","KV Cache Precision":"16-bit"}. This record does not assert independent reproduction.

### LogQuant

Math Performance: 63.31. Result ID: 6a0c1af0fbc373592c020fa2.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

Recorded evaluation configuration: {"Model":"Qwen1.5-14B-Chat-AWQ","KV Cache Precision":"2-bit"}. This record does not assert independent reproduction.

### KiVi

Math Performance: 59.82. Result ID: 6a0c1af0fbc373592c020fa1.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa744b1e209131cea60dab.md>)

Recorded evaluation configuration: {"Model":"Qwen1.5-14B-Chat-AWQ","KV Cache Precision":"2-bit"}. This record does not assert independent reproduction.

### 16-bit Baseline

Math Performance: 56.18. Result ID: 6a0c1af0fbc373592c020f9d.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

Recorded evaluation configuration: {"Model":"Qwen1.5-7B-Chat-AWQ","KV Cache Precision":"16-bit"}. This record does not assert independent reproduction.

### 16-bit Baseline

Math Performance: 52.99. Result ID: 6a0c1af0fbc373592c020fa3.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

Recorded evaluation configuration: {"Model":"Qwen2-7B-Instruct","KV Cache Precision":"16-bit"}. This record does not assert independent reproduction.

### LogQuant

Math Performance: 51.86. Result ID: 6a0c1af0fbc373592c020fa8.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

Recorded evaluation configuration: {"Model":"Phi-3-mini-128k-instruct","KV Cache Precision":"2-bit"}. This record does not assert independent reproduction.

### LogQuant

Math Performance: 49.28. Result ID: 6a0c1af0fbc373592c020f9f.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

Recorded evaluation configuration: {"Model":"Qwen1.5-7B-Chat-AWQ","KV Cache Precision":"2-bit"}. This record does not assert independent reproduction.

### LogQuant

Math Performance: 40.41. Result ID: 6a0c1af0fbc373592c020f9c.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

Recorded evaluation configuration: {"Model":"llama-3.1-8B-Instruct","KV Cache Precision":"2-bit"}. This record does not assert independent reproduction.

### KiVi

Math Performance: 39.27. Result ID: 6a0c1af0fbc373592c020f9e.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa744b1e209131cea60dab.md>)

Recorded evaluation configuration: {"Model":"Qwen1.5-7B-Chat-AWQ","KV Cache Precision":"2-bit"}. This record does not assert independent reproduction.

### LogQuant

Math Performance: 34.34. Result ID: 6a0c1af0fbc373592c020fa5.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

Recorded evaluation configuration: {"Model":"Qwen2-7B-Instruct","KV Cache Precision":"2-bit"}. This record does not assert independent reproduction.

### KiVi

Math Performance: 18.04. Result ID: 6a0c1af0fbc373592c020f9b.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa744b1e209131cea60dab.md>)

Recorded evaluation configuration: {"Model":"llama-3.1-8B-Instruct","KV Cache Precision":"2-bit"}. This record does not assert independent reproduction.

### KiVi

Math Performance: 12.59. Result ID: 6a0c1af0fbc373592c020fa7.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa744b1e209131cea60dab.md>)

Recorded evaluation configuration: {"Model":"Phi-3-mini-128k-instruct","KV Cache Precision":"2-bit"}. This record does not assert independent reproduction.

### KiVi

Math Performance: 3.71. Result ID: 6a0c1af0fbc373592c020fa4.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77a11e209131cea747e1.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa744b1e209131cea60dab.md>)

Recorded evaluation configuration: {"Model":"Qwen2-7B-Instruct","KV Cache Precision":"2-bit"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
