Character description generation on BookWorm
22.05PRISMAGPT-4.1 mini (Full context)
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| GPT-4.1 mini (Full context)Model=GPT-4.1 mini, Context=Full context, Trace=none, Protocol=Zero-Shot2026.04 | 22.05 | 16.67 | 75.53 | 35.41 | 17.71 | |
| Qwen-3-8b (Hierarchical-16k)Model=Qwen-3-8b, Context=Hierarchical-16k, Trace=none, Protocol=Zero-Shot2026.04 | 19.99 | 15.79 | 70.57 | 36.56 | 18.22 | |
| Qwen-3-8b (Hierarchical-16k + guided-QA)Model=Qwen-3-8b, Context=Hierarchical-16k, Trace=ZS, Protocol=Zero-Shot2026.04 | 19.82 | 16.2 | 70.74 | 37.67 | 17.9 | |
| Qwen-3-8b (Hierarchical-16k + reasoning)Model=Qwen-3-8b, Context=Hierarchical-16k, Trace=built-in, Protocol=Zero-Shot2026.04 | 19.74 | 15.16 | 71.15 | 35.82 | 17.33 | |
| Qwen-3-8b (Coref-32k + guided-QA)Model=Qwen-3-8b, Context=Coref-32k, Trace=GRPO, Protocol=Zero-Shot2026.04 | 19.49 | 14.86 | 51.23 | 37.66 | 18.01 | |
| Qwen-3-8b (Coref-32k + guided-QA)Model=Qwen-3-8b, Context=Coref-32k, Trace=GRPO, Protocol=SFT2026.04 | 19.32 | 15.1 | 52 | 37.58 | 18.42 | |
| Qwen-3-8b (Lead-128k w/ YaRN + guided-QA)Model=Qwen-3-8b, Context=Lead-128k (w/ YaRN), Trace=ZS, Protocol=Zero-Shot2026.04 | 19.32 | 15.77 | 71.55 | 37.8 | 17.77 | |
| Qwen-3-8b (Lead-128k w/ YaRN + guided-QA)Model=Qwen-3-8b, Context=Lead-128k (w/ YaRN), Trace=GRPO, Protocol=Zero-Shot2026.04 | 19.3 | 16.22 | 70.51 | 37.12 | 18.39 | |
| Qwen-3-8b (Coref-32k)Model=Qwen-3-8b, Context=Coref-32k, Trace=none, Protocol=Zero-Shot2026.04 | 19.11 | 13.96 | 52.58 | 36.12 | 18.44 | |
| Qwen-3-8b (Hierarchical-16k + guided-QA)Model=Qwen-3-8b, Context=Hierarchical-16k, Trace=GRPO, Protocol=Zero-Shot2026.04 | 18.94 | 15.36 | 67.92 | 36.49 | 18.19 | |
| Qwen-3-8b (Coref-32k + guided-QA)Model=Qwen-3-8b, Context=Coref-32k, Trace=ZS, Protocol=Zero-Shot2026.04 | 18.63 | 15.01 | 50.6 | 36.95 | 17.5 | |
| Qwen-3-8b (Coref-32k + reasoning)Model=Qwen-3-8b, Context=Coref-32k, Trace=built-in, Protocol=Zero-Shot2026.04 | 18.59 | 13.94 | 49.94 | 35.88 | 16.54 | |
| Qwen-3-8b (BM25-32k + guided-QA)Model=Qwen-3-8b, Context=BM25-32k, Trace=GRPO, Protocol=Zero-Shot2026.04 | 18.38 | 15.18 | 59.9 | 37.05 | 17.59 | |
| Qwen-3-8b (BM25-32k + guided-QA)Model=Qwen-3-8b, Context=BM25-32k, Trace=GRPO, Protocol=SFT2026.04 | 18.27 | 15.15 | 60.46 | 36.6 | 17.87 | |
| Qwen-3-8b (BM25-32k + guided-QA)Model=Qwen-3-8b, Context=BM25-32k, Trace=ZS, Protocol=Zero-Shot2026.04 | 17.84 | 14.66 | 59.96 | 36.23 | 17.43 | |
| Qwen-3-8b (Incremental-16k)Model=Qwen-3-8b, Context=Incremental-16k, Trace=none, Protocol=Zero-Shot2026.04 | 17.64 | 14.42 | 69.68 | 35.59 | 17.58 | |
| Qwen-3-8b (Coref-32k)Model=Qwen-3-8b, Context=Coref-32k, Trace=none, Protocol=SFT2026.04 | 17.46 | 14.36 | 52.54 | 35.59 | 19.26 | |
| GPT-4.1 mini (No context)Model=GPT-4.1 mini, Context=No context, Trace=none, Protocol=Zero-Shot2026.04 | 17.36 | 12.49 | — | 29.35 | 17.41 | |
| Qwen-3-8b (BM25-32k)Model=Qwen-3-8b, Context=BM25-32k, Trace=none, Protocol=Zero-Shot2026.04 | 17.09 | 13.56 | 58.58 | 34.06 | 18.18 | |
| Qwen-3-8b (BM25-32k + reasoning)Model=Qwen-3-8b, Context=BM25-32k, Trace=built-in, Protocol=Zero-Shot2026.04 | 17.06 | 13.5 | 58.44 | 35.13 | 16.51 | |
| Qwen-3-8b (Lead-128k w/ YaRN)Model=Qwen-3-8b, Context=Lead-128k (w/ YaRN), Trace=none, Protocol=Zero-Shot2026.04 | 17.02 | 13.68 | 69.29 | 34.94 | 18.11 | |
| Qwen-3-8b (Lead-128k w/ YaRN + reasoning)Model=Qwen-3-8b, Context=Lead-128k (w/ YaRN), Trace=built-in, Protocol=Zero-Shot2026.04 | 16.93 | 13.65 | 67.17 | 34.2 | 16.43 | |
| Qwen-3-8b (Incremental-16k + guided-QA)Model=Qwen-3-8b, Context=Incremental-16k, Trace=ZS, Protocol=Zero-Shot2026.04 | 16.8 | 14.56 | 68.2 | 34.92 | 16.98 | |
| Qwen-3-8b (Incremental-16k + guided-QA)Model=Qwen-3-8b, Context=Incremental-16k, Trace=GRPO, Protocol=Zero-Shot2026.04 | 16.59 | 13.63 | 66.05 | 34.51 | 17.42 | |
| Qwen-3-8b (BM25-32k)Model=Qwen-3-8b, Context=BM25-32k, Trace=none, Protocol=SFT2026.04 | 16.17 | 13.26 | 60 | 33.11 | 18.99 | |
| Qwen-3-8b (Incremental-16k + reasoning)Model=Qwen-3-8b, Context=Incremental-16k, Trace=built-in, Protocol=Zero-Shot2026.04 | 16.03 | 13.36 | 66.46 | 33.37 | 15.76 | |
| Qwen-3-8b (Lead-32k)Model=Qwen-3-8b, Context=Lead-32k, Trace=none, Protocol=Zero-Shot2026.04 | 15.72 | 12.35 | 61.01 | 30.6 | 17.79 | |
| Qwen-3-8b (No Context)Model=Qwen-3-8b, Context=No Context, Trace=none, Protocol=Zero-Shot2026.04 | 11.97 | 10.31 | — | 25.97 | 17.55 |