Question Answering on NQ (EM, F1, Token Count)
41.88Exact Match (EM)CORE (1.5B)
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| CORE (1.5B)Compression setting=Generalization to top 10 documents, Training setting=Trained on top 5 documents, Method=CORE (1.5B)2025.08 | 41.88 | 51.26 | 52 | |
| CORE (1.5B)Compression setting=Compression of top 5 documents, Method=CORE (1.5B)2025.08 | 41.02 | 50.4 | 46 | |
| Top 10 DocumentsCompression=None, Retrieval=Top 10 Documents2025.08 | 38.67 | 50.03 | 1,425 | |
| Top 5 DocumentsCompression=None, Retrieval=Top 5 Documents2025.08 | 38.03 | 49.16 | 712 | |
| Top 3 DocumentsCompression=None, Retrieval=Top 3 Documents2025.08 | 37.78 | 48.45 | 427 | |
| DeepSeek-V3 (671B)Compression setting=Compression of top 5 documents, Method=DeepSeek-V3 (671B)2025.08 | 37.73 | 50.39 | 54 | |
| QGC (1.5B)Compression setting=Compression of top 5 documents, Method=QGC (1.5B)2025.08 | 36.23 | 45.88 | 49 | |
| NoiseFilter-IB (1.5B)Compression setting=Compression of top 5 documents, Method=NoiseFilter-IB (1.5B)2025.08 | 35.15 | 45.94 | 48 | |
| Top 1 DocumentCompression=None, Retrieval=Top 1 Document2025.08 | 34.46 | 44.41 | 142 | |
| RECOMP-Abs (1.5B)Compression setting=Compression of top 5 documents, Method=RECOMP-Abs (1.5B)2025.08 | 34.18 | 46.26 | 58 | |
| RECOMP-Ext (1.5B)Compression setting=Compression of top 5 documents, Method=RECOMP-Ext (1.5B)2025.08 | 33.84 | 46.05 | 56 | |
| LongLLMLingua (1.5B)Compression setting=Compression of top 5 documents, Method=LongLLMLingua (1.5B)2025.08 | 33.65 | 43.15 | 152 | |
| Qwen2.5-1.5B-InstructCompression setting=Compression of top 5 documents, Method=Qwen2.5-1.5B-Instruct2025.08 | 31.94 | 43.03 | 36 | |
| BM25Compression setting=Compression of top 5 documents, Method=BM252025.08 | 25.23 | 36.47 | 37 | |
| No RetrievalRetrieval=None2025.08 | 21.36 | 30.97 | 0 |