Document Question Answering on Qasper
40.8AccuracyFP8
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FP8Backbone=Llama 3.1 8B, Compression Ratio (CR)=2, Evaluation Protocol=2-shot2025.11 | 40.8 | — | |
| kvtc16×Backbone=Llama 3.1 8B, Compression Ratio (CR)=18-22, Evaluation Protocol=2-shot2025.11 | 40.7 | — | |
| VanillaBackbone=Llama 3.1 8B, Compression Ratio (CR)=1, Evaluation Protocol=2-shot2025.11 | 40.4 | — | |
| GEAR2-bitBackbone=Llama 3.1 8B, Compression Ratio (CR)=5, Evaluation Protocol=2-shot2025.11 | 40.4 | — | |
| kvtc8×Backbone=Llama 3.1 8B, Compression Ratio (CR)=9-10, Evaluation Protocol=2-shot2025.11 | 40.1 | — | |
| kvtc32×Backbone=Llama 3.1 8B, Compression Ratio (CR)=34-44, Evaluation Protocol=2-shot2025.11 | 39.4 | — | |
| KIVI2-bitBackbone=Llama 3.1 8B, Compression Ratio (CR)=5, Evaluation Protocol=2-shot2025.11 | 39.1 | — | |
| kvtc8×Backbone=MN-Minitron 8B, Compression Ratio (CR)=10-11, Evaluation Protocol=2-shot2025.11 | 39.1 | — | |
| TOVABackbone=Llama 3.1 8B, Compression Ratio (CR)=8, Evaluation Protocol=2-shot2025.11 | 38.6 | — | |
| kvtc16×Backbone=MN-Minitron 8B, Compression Ratio (CR)=17-21, Evaluation Protocol=2-shot2025.11 | 38.6 | — | |
| GEAR2-bitBackbone=Mistral NeMo 12B, Compression Ratio (CR)=5, Evaluation Protocol=2-shot2025.11 | 38.6 | — | |
| VanillaBackbone=Mistral NeMo 12B, Compression Ratio (CR)=1, Evaluation Protocol=2-shot2025.11 | 38.4 | — | |
| FP8Backbone=MN-Minitron 8B, Compression Ratio (CR)=2, Evaluation Protocol=2-shot2025.11 | 38.3 | — | |
| VanillaBackbone=MN-Minitron 8B, Compression Ratio (CR)=1, Evaluation Protocol=2-shot2025.11 | 38.2 | — | |
| GEAR2-bitBackbone=MN-Minitron 8B, Compression Ratio (CR)=5, Evaluation Protocol=2-shot2025.11 | 38.2 | — | |
| KIVI2-bitBackbone=MN-Minitron 8B, Compression Ratio (CR)=5, Evaluation Protocol=2-shot2025.11 | 38.2 | — | |
| KIVI2-bitBackbone=Mistral NeMo 12B, Compression Ratio (CR)=5, Evaluation Protocol=2-shot2025.11 | 38.2 | — | |
| kvtc64×Backbone=MN-Minitron 8B, Compression Ratio (CR)=53-95, Evaluation Protocol=2-shot2025.11 | 38.1 | — | |
| kvtc64×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=51-87, Evaluation Protocol=2-shot2025.11 | 38 | — | |
| FP8Backbone=Mistral NeMo 12B, Compression Ratio (CR)=2, Evaluation Protocol=2-shot2025.11 | 37.9 | — | |
| kvtc64×Backbone=Llama 3.1 8B, Compression Ratio (CR)=60-88, Evaluation Protocol=2-shot2025.11 | 37.8 | — | |
| kvtc32×Backbone=MN-Minitron 8B, Compression Ratio (CR)=32-46, Evaluation Protocol=2-shot2025.11 | 37.7 | — | |
| kvtc8×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=10-11, Evaluation Protocol=2-shot2025.11 | 37.6 | — | |
| kvtc16×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=17-20, Evaluation Protocol=2-shot2025.11 | 37.6 | — | |
| kvtc32×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=31-43, Evaluation Protocol=2-shot2025.11 | 37.5 | — | |
| TOVABackbone=Mistral NeMo 12B, Compression Ratio (CR)=8, Evaluation Protocol=2-shot2025.11 | 36 | — | |
| xKVBackbone=Llama 3.1 8B, Compression Ratio (CR)=1-5, Evaluation Protocol=2-shot2025.11 | 35.6 | — | |
| xKVBackbone=MN-Minitron 8B, Compression Ratio (CR)=1-5, Evaluation Protocol=2-shot2025.11 | 34.5 | — | |
| H2OBackbone=Llama 3.1 8B, Compression Ratio (CR)=8, Evaluation Protocol=2-shot2025.11 | 34.3 | — | |
| TOVABackbone=MN-Minitron 8B, Compression Ratio (CR)=8, Evaluation Protocol=2-shot2025.11 | 33.9 | — | |
| xKVBackbone=Mistral NeMo 12B, Compression Ratio (CR)=1-5, Evaluation Protocol=2-shot2025.11 | 33.5 | — | |
| H2OBackbone=MN-Minitron 8B, Compression Ratio (CR)=8, Evaluation Protocol=2-shot2025.11 | 30 | — | |
| H2OBackbone=Mistral NeMo 12B, Compression Ratio (CR)=8, Evaluation Protocol=2-shot2025.11 | 29.5 | — | |
| BookRAGBaseline Type=Our proposed2025.12 | 0.552 | 0.611 | |
| DocETLBaseline Type=Layout segmented RAG2025.12 | 0.423 | 0.504 | |
| Layout + VanillaBaseline Type=Conventional RAG2025.12 | 0.407 | 0.446 | |
| Vanilla RAGBaseline Type=Conventional RAG2025.12 | 0.406 | 0.444 | |
| RAPTORBaseline Type=Graph-based RAG2025.12 | 0.394 | 0.441 | |
| BM25Baseline Type=Conventional RAG2025.12 | 0.381 | 0.425 | |
| GraphRAG-LocalBaseline Type=Graph-based RAG2025.12 | 0.359 | 0.392 | |
| GraphRankerBaseline Type=Layout segmented RAG2025.12 | 0.329 | 0.376 | |
| MM-VanillaBaseline Type=Layout segmented RAG2025.12 | 0.279 | 0.293 | |
| Tree-TraverseBaseline Type=Layout segmented RAG2025.12 | 0.273 | 0.321 | |
| GraphRAG-GlobalBaseline Type=Graph-based RAG2025.12 | 0.24 | 0.241 |