Information Extraction on ACL Anthology gold benchmark 100 query–chunk pairs
65.43Word Precisionacl-verbatim-modernbert
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| acl-verbatim-modernbertNumber of Parameters=150M, Threshold=0.2, Min-span length=10, Merge gap=20, Hardware=CPU2026.05 | 65.43 | 45.43 | 53.63 | 27.27 | 31.47 | 48.22 | 35.9 | 37.18 | 42.31 | 0.47 | |
| verbatim-rag-modern-bert-v2Number of Parameters=150M, Threshold=0.2, Min-span length=30, Merge gap=20, Hardware=CPU2026.05 | 63 | 36.58 | 46.29 | 30.3 | 31.82 | 46.97 | 26.92 | 32.05 | 38.46 | 0.4 | |
| Zilliz Semantic HighlightThreshold=0.3, Hardware=CPU2026.05 | 46.97 | 22.11 | 30.07 | 33.33 | 35.46 | 39.53 | 2.56 | 8.97 | 19.23 | 1.04 | |
| Qwen 3.6 35BExtraction Prompt=Default2026.05 | 44.74 | 40.88 | 42.72 | 49.05 | 50.95 | 53.33 | 15.38 | 25.64 | 41.03 | 1.63 | |
| GLM-5Extraction Prompt=Default2026.05 | 44.5 | 53.8 | 48.71 | 52.87 | 54.25 | 58.36 | 20.51 | 34.62 | 53.85 | 1.04 | |
| Mistral Small 2603Extraction Prompt=Default2026.05 | 40.41 | 55.99 | 46.94 | 43.86 | 46.24 | 49.73 | 17.95 | 39.74 | 57.69 | 0.78 | |
| Qwen 3.6 35BExtraction Prompt=Paragraph-oriented2026.05 | 39.43 | 57.35 | 46.73 | 34.78 | 36.96 | 47.83 | 44.87 | 51.28 | 61.54 | 1.2 | |
| Mistral Small 2603Extraction Prompt=Paragraph-oriented2026.05 | 34.22 | 73.03 | 46.61 | 27.7 | 33.61 | 39.04 | 48.72 | 65.38 | 75.64 | 1.07 | |
| Nemotron-120B-A12BExtraction Prompt=Default2026.05 | 34.14 | 50.92 | 40.88 | 36.62 | 36.62 | 42.88 | 43.59 | 43.59 | 52.56 | 0.38 | |
| Nemotron-120B-A12BExtraction Prompt=Paragraph-oriented2026.05 | 30.15 | 62.72 | 40.73 | 29.93 | 31.29 | 44.9 | 58.97 | 62.82 | 62.82 | 0.54 | |
| ProvenceSetting=Native setting with internal splitting, Hardware=CPU2026.05 | 27.58 | 45.7 | 34.4 | 21.56 | 22.84 | 29.23 | 16.67 | 21.79 | 41.03 | 2.4 |